企业级大数据分析平台实践案例课件

1、企业级大数据分析平台实践案例企业级大数据解决方案企业级hadoop高可用HDFS集群企业级大数据分析平台Hive企业级大数据数据仓库Hbase企业级数据流实时导入工具Flume企业级关系数据库迁移工具sqoopSqoop根据数据库表结构自动创建class文件，提交到mapreduce框架上运行大数据生态链的常用工具pig - 精短强悍的数据整理清洗工具spark - 基于内存的流式数据分析工具，内置机器学习库oozie - 任务自动化调度工具kafka - 跨平台的数据传输工具，支持各种传输协议和加密压缩等功能impala - 类似hive的数据分析工具，支持SQL查询语言，速度更快tez -

2、 优化mapreduce计算路径的计算框架kudu - 一种更快速的数据分析平台solr - 企业级搜索引擎企业级hadoop集群搭建准备准备至少3台联网的主机，单个主机至少4G内存，10G磁 盘剩余空间。分别安装Hanwate_Bigdata_OS_7_Firefly操作系统，该系统 整合了大数据生态链的常用工具。按照后面的向导，分别在每台主机上安装相应的软件组件，并配 置其角色，启动相应的服务。分布式hadoop集群的主机角色主机名IP地址角色组件master1192.168.X.3masterNamenode Datanode NodeManagermaster2192.168.X.4m

3、asterSecondaryNamenode ResourceManager Datanode NodeManagerslave1192.168.X.5slaveJobHistoryServer Datanode NodeManager伪分布式hadoop集群的主机角色主机名IP地址角色组件master1192.168.X.3masterNamenode Datanode NodeManagermaster2192.168.X.3masterSecondaryNamenode ResourceManager Datanode NodeManagerslave1192.168.X.3slaveJ

4、obHistoryServer Datanode NodeManager修改主机地址和主机名每台主机分别修改IP地址和主机名，例如在master1上: 修改网卡配置文件/etc/sysconfig/network-scripts/ifcfg-xxxBOOTPROTO=none IPADDR=191.168.X.3 GATEWAY=192.168.X.1 NETMASK=主机名配置文件为/etc/hostname，例如在master1上内容为：master1配置主机名映射在所有节点上修改/etc/hosts:（文件末尾追加）191.168.X.3191.168.X.41

5、91.168.X.5master1 master2 slave1伪分布式集群的配置：191.168.X.3master1 master2 slave1应用配置文件所有主机同步/etc/hosts重新启动服务器# reboot检查主机名和其他节点是否可以访问# for host in master1 master2 slave1 doping -c1 $hostdone配置JAVA环境变量在/etc/profile 文件末尾追加export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-51- 1.b12.el7_4.x86_64/jreexport CLA

6、SSPATH=.:$CLASSPATH:$JAVA_HOME/lib export PATH=$PATH:$JAVA_HOME/bin保存文件后执行命令使文件生效#source/etc/profile集群搭建根据主机角色在不同的节点上安装软件根据集群规划对各个节点上的服务进行安装和配置配置集群所需的配置文件（所有节点必须配置而且保持一致）- 配置文件路径为/etc/hadoop/conf配置集群slaves增加slave节点# vim/etc/hadoop/conf/slaves配置集群core-site.xmlNameValue作用fs.defaultFShdfs:/master1:8020

7、HDFS的访问入口hadoop.tmp.dir/usr/hdp/tmphadoop文件系统依赖的基础配置配置集群 hdfs-site.xmlNameValue作用.dir/hadoop/hdfs/namenamenode数据的存放位置dfs.datanode.data.dir/hadoop/hdfs/datadatanode数据块的存放位置dfs.replication3hdfs的文件备份系数，伪分布式集群只能为1创建HDFS需要用到的Linux目录# mkdir -p /usr/hdp/tmp# mkdir -p /hadoop/hdfs/data,name # chown -R hdfs:

8、hadoop/hadoop# chown -R hdfs:hadoop/usr/hdp/tmp初始化hdfs文件系统在master1 上操作：# sudo -u hdfs$ hdfs namenode -format以hdfs账户初始化hdfs文件系统启动hdfs文件系统启动master1节点上的服务：# systemctl start hadoop-hdfs-namenode # systemctl start hadoop-hdfs-datanode启动master2节点上的服务：# systemctl start hadoop-hdfs-secondarynamenode # syste

9、mctl start hadoop-hdfs-datanode启动slave1节点上的服务：# systemctl start hadoop-hdfs-datanode验证方式可以通过systemctlstatusxxx 来查看服务的状态以root身份运行jps分别会看到master1显示：master2显示：slave1显示：访问NameNode WEB后台，比如:50070为普通用户创建hdfs工作目录使用linux管理员创建新用户# useradd hadoop切换成HDFS管理员# su - hdfs创建普通用户的目录$ hadoop fs -mkdir -p /user/hadoop

10、$ hadoop fs -chown hadoop /user/hadoop$ exit使用普通用户验证hdfs工作目录# su - hadoop$ hadoop fs -mkdir input$ hadoop fs -ls软件排错参考日志hdfs组件的日志/var/log/hadoop-hdfs/*.log/var/log/hadoop-hdfs/*.outyarn组件的日志/var/log/hadoop-yarn/*.log/var/log/hadoop-yarn/*.outmapreduce组件的日志/var/log/hadoop-mapreduce/*.log/var/log/hado

11、op-mapreduce/*.out准备运行分布式任务的目录# su hdfs$ hadoop fs -mkdir /tmp$ hadoop fs -chmod 1777 /tmp$ hadoop fs -mkdir -p /var/log/hadoop-yarn$ hadoop fs -chown yarn:mapred /var/log/hadoop-yarn$ hadoop fs -mkdir /user/history$ hadoop fs -chmod 1777 /user/history$ hadoop fs -chown mapred:hadoop /user/history配置

12、yarn-site.xmlNameValue作用yarn.resourcemanager.hostnamemaster2指定yarn任务管理器的入 口yarn.nodemanager.aux-servicesmapreduce_shuffle为mapreduce分配yarn服 务yarn.nodemanager.local-dirsfile:/hadoop/yarn/localnodemanager本机计算任 务的临时文件yarn.nodemanager.log-dirs/var/log/hadoop- yarn/containersnodemanager日志输出yarn.nodemanage

13、r.remote-app- log-dir/var/log/hadoop-yarn/apps远程任务的输出yarn.log-aggregation-enabletrue日志汇集配置yarn-site.xml。NameValue作用yarn.scheduler.minimum-allocation-mb511单个任务可申请的最小 内存资源量yarn.scheduler.maximum-allocation-mb2049单个任务可申请的最大 内存资源量yarn.nodemanager.vmem-pmem-ratio4每使用1MB物理内存， 最多可用的虚拟内存数yarn.nodemanager.vm

14、em-check-enabledfalse针对虚拟内存监控的开 关注：以上设置内存值情况为MR任务因内存问题崩溃即可据此设置，值不唯一配置yarn-site.xmlNameValue作用yarn.application.classpath$HADOOP_CONF_DIR,yarn运行所需环/usr/hdp/-235/hadoop/*,境变量/usr/hdp/-235/hadoop/lib/*,/usr/hdp/-235/hadoop-hdfs/*,/usr/hdp/-235/hadoop-hdfs/lib/*,/usr/hdp/-235/hadoop-yarn/*,/usr/hdp/-235/

15、hadoop-yarn/lib/*,/usr/hdp/-235/hadoop-mapreduce/*,/usr/hdp/-235/hadoop-mapreduce/lib/*,/usr/hdp/-235/hadoop-httpfs/*,/usr/hdp/-235/hadoop-httpfs/lib/*配置mapred-site.xmlNameValue作用yarn分配mapreduce框架mapreduce.jobhistory.addressslave1:10020historyserver地址mapreduce.jobhistory.webapps.addressslave1:19888H

16、istoryserver web端口yarn.app.mapreduce.am.staging-dir/useryarn任务的临时输出目录mapreduce.application.classpath(较长，见上一页 yarn.application.classpath)MapReduce需要的环境变 量配置mapred-site.xmlNameValue作用mapreduce.map.java.opts-Xmx1024M运行 Map 任务的 JVM 参数mapreduce.map.memory.mb31Container 这个进程的最大可用 内存大小mapreduce.reduce.java

17、.opts-Xmx1024M运行 Reduce 任务的 JVM 参数mapreduce.reduce.memory.mb63Container 这个进程的最大可用 内存大小注：以上设置内存值情况为MR任务因内存问题卡死即可据此设置，值不唯一配置yarn的linux本地目录# touch /etc/hadoop/conf/yarn-env.sh# mkdir -p /hadoop/yarn/local# chown yarn:yarn -R /hadoop/yarn/local启动服务在master2上开启resourcemanager# systemctl start hadoop-yarn-

18、resourcemanager(访问WEB后台http:/master2:8088)(若从windows系统访问该主 机名不可用，在windows的hosts文件中加入主机地址映射即可)在slave1上开启historyserver# systemctl start hadoop-mapreduce-historyserver（访问WEB后台http:/slave1:19888)所有datanode节点上开nodemanager# systemctl start hadoop-yarn-nodemanager测试mapreduce任务# su - hdfs$ cd /etc/hadoop/co

19、nf$ hadoop fs -mkdir-p/user/hdfs/input$ hadoop fs -put*/user/hdfs/input$ hadoop jar /usr/hdp/-235/hadoop- mapreduce/hadoop-mapreduce-examples-.6.3.0- 235.jar grep input output value运行成功的输出结果Yarn WEB后台的任务状态配置 flume同步集群时间配置集群时钟同步，在一台服务器上配置 /etc/chrony.conf，打开allow 191.168/16 bindcmdaddress 在其他服务器上配置 /

20、etc/chrony.conf: server iburst都需要重新启动 chronyd服务 # systemctl restart chrony配置代理a1.sources = r1 a1.sinks = k1 a1.channels = c1a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444a1.sinks.k1.type = loggera1.channels.c1.type = memory a1.channels.c1.capacity = 1000a1.chan

21、nels.c1.transactionCapacity = 100a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1启动代理上述配置文件存入/etc/flume/conf/flume.conf启动agent# flume-ngagent-conf /etc/flume/conf-conf-file/etc/flume/conf/flume.conf-namea1-Dflume.root.logger=INFO,console启动代理系统提示连接到44444后，打开另一终端进行测试# nc localhost 44444出现ok的同时，另一终

22、端接受到信息，并显示flume 的部署Flume SourcesAvroThriftExecJMSSpooling DirectoryTaildirKafkaNetcat(tcp & udp)SyslogHttpStressSourceScribeavro 源的配置a1.sources = r1 a1.channels = c1 a1.sources.r1.type = avro a1.sources.r1.channels = c1 a1.sources.r1.bind = a1.sources.r1.port = 4141thrift 源a1.sources = r1 a1.channel

23、s = c1 a1.sources.r1.type = thrift a1.sources.r1.channels = c1 a1.sources.r1.bind = a1.sources.r1.port = 4141exec 源a1.sources = r1 a1.channels = c1 a1.sources.r1.type = execmand = tail -F /var/log/secure a1.sources.r1.channels = c1netcat 源a1.sources = r1 a1.channels = c1 a1.sources.r1.type = netcata

24、1.sources.r1.bind = a1.sources.r1.port = 6666 a1.sources.r1.channels = c1syslog 源a1.sources = r1 a1.channels = c1a1.sources.r1.type = syslogtcp a1.sources.r1.port = 5140 a1.sources.r1.host = localhost a1.sources.r1.channels = c1http 源a1.sources = r1 a1.channels = c1 a1.sources.r1.type = http a1.sour

25、ces.r1.port = 5140 a1.sources.r1.channels = c1a1.sources.r1.handler = org.example.rest.RestHandler a1.sources.r1.handler.nickname = random propsFlume ChannelsMemoryJDBCKafkaFileMemory Channela1.channels = c1 a1.channels.c1.type = memory a1.channels.c1.capacity = 10000a1.channels.c1.transactionCapaci

26、ty = 10000a1.channels.c1.byteCapacityBufferPercentage = 20a1.channels.c1.byteCapacity = 800000File Channela1.channels = c1 a1.channels.c1.type = filea1.channels.c1.checkpointDir = /mnt/flume/checkpoint a1.channels.c1.dataDirs = /mnt/flume/dataFlume SinKsHDFSHiveLoggerAvroThriftIRCFile RollNullHbaseS

27、inkAsyncHBaseSinkMorphlineSolrSinkElasticSearchSinkKafkaHTTPHDFS Sinka1.channels = c1 a1.sinks = k1 a1.sinks.k1.type = hdfs a1.sinks.k1.channel = c1a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S a1.sinks.k1.hdfs.filePrefix = events-a1.sinks.k1.hdfs.round = true a1.sinks.k1.hdfs.roundValue =

28、10 a1.sinks.k1.hdfs.roundUnit = minuteHive Sinka1.channels = c1 a1.channels.c1.type = memory a1.sinks = k1a1.sinks.k1.type = hive a1.sinks.k1.channel = c1a1.sinks.k1.hive.metastore = thrift:/:9083 a1.sinks.k1.hive.database = logsdb a1.sinks.k1.hive.table = weblogsa1.sinks.k1.hive.partition = asia,%c

29、ountry,%y-%m-%d-%H-%MHive Sink（续）a1.sinks.k1.useLocalTimeStamp = false a1.sinks.k1.round = true a1.sinks.k1.roundValue = 10 a1.sinks.k1.roundUnit = minute a1.sinks.k1.serializer = DELIMITED a1.sinks.k1.serializer.delimiter = t a1.sinks.k1.serializer.serdeSeparator = t a1.sinks.k1.serializer.fieldnam

30、es =id,msgLogger Sinka1.channels = c1 a1.sinks = k1 a1.sinks.k1.type = logger a1.sinks.k1.channel = c1Avro Sinka1.channels = c1 a1.sinks = k1 a1.sinks.k1.type = avro a1.sinks.k1.channel = c1a1.sinks.k1.hostname = 0a1.sinks.k1.port = 4545HTTP Sinka1.channels = c1 a1.sinks = k1 a1.sinks.k1.type = http

31、 a1.sinks.k1.channel = c1a1.sinks.k1.endpoint = http:/localhost:8080/someuri a1.sinks.k1.connectTimeout = 2000a1.sinks.k1.requestTimeout = 2000 a1.sinks.k1.acceptHeader = application/json a1.sinks.k1.contentTypeHeader = application/jsonHTTP Sink（续）a1.sinks.k1.defaultBackoff = true a1.sinks.k1.defaul

32、tRollback = true a1.sinks.k1.defaultIncrementMetrics = false a1.sinks.k1.backoff.4XX = false a1.sinks.k1.rollback.4XX = false a1.sinks.k1.incrementMetrics.4XX = true a1.sinks.k1.backoff.200 = false a1.sinks.k1.rollback.200 = false a1.sinks.k1.incrementMetrics.200 = trueflume 应用总结使用原则sink 负责写，写之前，目标地

33、址可访问channel 负责传输，若无审计需要，都是内存source 负责读，读取的数据是变化的案例使用 flume 收集 nginx 服务器的日志到 hdfs 中参考配置文件配置文件/etc/flume/conf/flume.conf # #配置Agenta1.sources = r1 a1.sinks = k1 a1.channels = c1 # # 配置Sourcea1.sources.r1.type = exec a1.sources.r1.channels = c1a1.sources.r1.deserializer.outputCharset = UTF-8 # # 配置需要监控的日志输出目录mand = tail -F /var/log/nginx/access.log（续）# # 配 置 Sink a1.sinks.k1.type = hdfs a1.sinks.k1.channel = c1a1.sinks.k1.hdfs.useLocalTimeStamp = truea1.sinks.k1.hdfs.path = hdfs:/master1/user/hdf