01Hadoop-2.7.2+Zookeeper-3.4.6完全分布式环境搭建(HDFS,YARN HA).docx

Hadoop-2.7.2+Zookeeper-3.4.6完全分布式环境搭建一.版本组件名 版本 说明 JRE java version 1.7.0_67Java(TM) SE Runtime Environment (build 1.7.0_67-b01)Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)Hadoop hadoop-2.7.2.tar.gz主程序包 Zookeeper zookeeper-3.4.6.tar.gz热切,Yarn 存储数据使用的协调服务 二.主机规划IP Host 及安装软件部署模块 进程 5sht-sgmhadoopnn-01hadoopNameNode ResourceManager NameNode DFSZKFailoverController ResourceManager 6sht-sgmhadoopnn-02hadoopNameNode ResourceManager NameNode DFSZKFailoverController ResourceManager 8sht-sgmhadoopdn-01hadoop、zookeeperDataNode NodeManager Zookeeper DataNode NodeManager JournalNode QuorumPeerMain 9sht-sgmhadoopdn-02Hadoop、zookeeperDataNode NodeManager Zookeeper DataNode NodeManager JournalNode QuorumPeerMain 0sht-sgmhadoopdn-03Hadoop、zookeeperDataNode NodeManager Zookeeper DataNode NodeManager JournalNode QuorumPeerMain 三.目录规划名称 路径 $HADOOP_HOME /hadoop/hadoop-2.7.2 Data $ HADOOP_HOME/data Log $ HADOOP_HOME/logs 四.常用脚本及命令1.启动集群start-dfs.shstart-yarn.sh2.关闭集群stop-yarn.shstop-dfs.sh3.监控集群hdfs dfsadmin -report4.单个进程启动/关闭hadoop-daemon.sh start|stop namenode|datanode| journalnodeyarn-daemon.sh start |stop resourcemanager|nodemanager/uid-25723371-id-4943894.html五.环境准备1 .设置ip地址(5台) rootsht-sgmhadoopnn-01 # vi /etc/sysconfig/network-scripts/ifcfg-eth0DEVICE=eth0BOOTPROTO=staticDNS1=3DNS2=4GATEWAY=HWADDR=00:50:56:82:50:1EIPADDR=5NETMASK=NM_CONTROLLED=yesONBOOT=yesTYPE=EthernetUUID=257c075f-6c6a-47ef-a025-e625367cbd9c执行命令: service network restart验证:ifconfig2 .关闭防火墙(5台) 执行命:service iptables stop验证:service iptables status3.关闭防火墙的自动运行(5台) 执行命令:chkconfig iptables off验证:chkconfig -list | grep iptables4 设置主机名(5台) 执行命令(1)hostname sht-sgmhadoopnn-01(2)vi /etc/sysconfig/networkrootsht-sgmhadoopnn-01 # vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=GATEWAY=5 ip与hostname绑定(5台) rootsht-sgmhadoopnn-01 # vi /etc/hosts localhost localhost.localdomain localhost4 localhost4.localdomain4:1 localhost localhost.localdomain localhost6 localhost6.localdomain65 sht-sgmhadoopnn-016 sht-sgmhadoopnn-028 sht-sgmhadoopdn-019 sht-sgmhadoopdn-020 sht-sgmhadoopdn-03验证:ping sht-sgmhadoopnn-016. 设置5台machines,SSH互相通信7 .安装JDK(5台)(1)执行命令 rootsht-sgmhadoopnn-01 # cd /usr/javarootsht-sgmhadoopnn-01 java# cp /tmp/jdk-7u67-linux-x64.gz ./rootsht-sgmhadoopnn-01 java# tar -xzvf jdk-7u67-linux-x64.gz (2)vi /etc/profile 增加内容如下: export JAVA_HOME=/usr/java/jdk1.7.0_67 export HADOOP_HOME=/hadoop/hadoop-2.7.2 export ZOOKEEPER_HOME=/hadoop/zookeeper export PATH=.:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ZOOKEEPER_HOME/bin:$PATH#先把HADOOP_HOME, ZOOKEEPER_HOME配置了 #本次实验机器已经配置好了jdk1.7.0_67-cloudera (3)执行 source /etc/profile(4)验证:java version8.创建文件夹(5台) mkdir /hadoop六.安装Zookeepersht-sgmhadoopdn-01/02/031.下载解压zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-01 tmp# wget /apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-02 tmp# wget /apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-03 tmp# wget /apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-01 tmp# tar -xvf zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-02 tmp# tar -xvf zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-03 tmp# tar -xvf zookeeper-3.4.6.tar.gzrootsht-sgmhadoopdn-01 tmp# mv zookeeper-3.4.6 /hadoop/zookeeperrootsht-sgmhadoopdn-02 tmp# mv zookeeper-3.4.6 /hadoop/zookeeperrootsht-sgmhadoopdn-03 tmp# mv zookeeper-3.4.6 /hadoop/zookeeper2.修改配置rootsht-sgmhadoopdn-01 tmp# cd /hadoop/zookeeper/confrootsht-sgmhadoopdn-01 conf# cp zoo_sample.cfg zoo.cfgrootsht-sgmhadoopdn-01 conf# vi zoo.cfg修改dataDirdataDir=/hadoop/zookeeper/data添加下面三行server.1=sht-sgmhadoopdn-01:2888:3888server.2=sht-sgmhadoopdn-02:2888:3888server.3=sht-sgmhadoopdn-03:2888:3888rootsht-sgmhadoopdn-01 conf# cd ./rootsht-sgmhadoopdn-01 zookeeper# mkdir datarootsht-sgmhadoopdn-01 zookeeper# touch data/myidrootsht-sgmhadoopdn-01 zookeeper# echo 1 data/myidrootsht-sgmhadoopdn-01 zookeeper# more data/myid1# sht-sgmhadoopdn-02/03,也修改配置,就如下不同rootsht-sgmhadoopdn-02 zookeeper# echo 2 data/myidrootsht-sgmhadoopdn-03 zookeeper# echo 3 data/myid七.安装Hadoop(HDFS HA+YARN HA)#step37,用SecureCRT ssh 到 linux的环境中,假如copy 内容从window 到 linux 中,中文乱码,请参照修改/qi09/archive/2013/02/05/2892922.html1.下载解压hadoop-2.7.2.tar.gzrootsht-sgmhadoopnn-01 tmp# wget /dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz -no-check-certificaterootsht-sgmhadoopnn-01 tmp# tar -xvf hadoop-2.7.2.tar.gzrootsht-sgmhadoopnn-01 tmp# mv /tmp/hadoop-2.7.2 /hadoop/hadoop-2.7.2rootsht-sgmhadoopnn-01 tmp# cd /hadoop/hadoop-2.7.2/etc/hadooprootsht-sgmhadoopnn-01 etc# pwd/hadoop/hadoop-2.7.2/etc/hadoop2.修改$HADOOP_HOME/etc/hadoop/hadoop-env.shexport JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera3.修改$HADOOP_HOME/etc/hadoop/core-site.xmlfs.defaultFShdfs:/erval14404.修改$HADOOP_HOME/etc/hadoop/.dir/hadoop/hadoop-2.7.2/data/dfs/name namenode 存放name table(fsimage)本地目录（需要修改）node.edits.dir$.dirnamenode粗放 transaction file(edits)本地目录（需要修改）dfs.datanode.data.dir/hadoop/hadoop-2.7.2/data/dfs/datadatanode存放block本地目录（需要修改）nodes.myclusternn1,node.rpc-address.mycluster.nn1sht-sgmhadoopnn-01:8020node.rpc-address.mycluster.nn2sht-sgmhadoopnn-02:8020node.http-address.mycluster.nn1sht-sgmhadoopnn-01:50070node.http-address.mycluster.nn2sht-sgmhadoopnn-02:50070dfs.journalnode.http-address:8480dfs.journalnode.rpc-address:8485!-格式：qjournal:/;/ 端口同journalnode.rpc-address -node.shared.edits.dirqjournal:/sht-sgmhadoopdn-01:8485;sht-sgmhadoopdn-02:8485;sht-sgmhadoopdn-03:8485/myclusterdfs.journalnode.edits.dir/hadoop/hadoop-2.7.2/data/dfs/node.ha.ConfiguredFailoverProxyProviderdfs.ha.fencing.methodssshfencedfs.ha.fencing.ssh.private-key-files/root/.ssh/id_rsadfs.ha.fencing.ssh.connect-timeout30000dfs.ha.automatic-failover.enabledtrueha.zookeeper.quorumsht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181ha.zookeeper.session-timeout.ms20005.修改$HADOOP_HOME/etc/hadoop/yarn-env.sh#Yarn Daemon Options#export YARN_RESOURCEMANAGER_OPTS#export YARN_NODEMANAGER_OPTS#export YARN_PROXYSERVER_OPTS#export HADOOP_JOB_HISTORYSERVER_OPTS#Yarn Logsexport YARN_LOG_DIR=/hadoop/hadoop-2.7.2/logs6.修改$HADOOP_HOEM/etc/hadoop/mapred-site.xmlrootsht-sgmhadoopnn-01 hadoop# cp mapred-site.xml.template mapred-site.xmlrootsht-sgmhadoopnn-01 hadoop# vi yarnmapreduce.jobhistory.address:10020mapreduce.jobhistory.webapp.address:198887.修改$HADOOP_HOME/etc/hadoop/yarn-site.xmlyarn.nodemanager.aux-servicesmapreduce_shuffleyarn.nodemanager.aux-services.mapreduce.shuffle.classorg.apache.hadoop.mapred.ShuffleHandlerAddress where the localizer IPC is.yarn.nodemanager.localizer.address:23344NM Webapp address.yarn.nodemanager.webapp.address:23999yarn.resourcemanager.connect.retry-interval.ms2000yarn.resourcemanager.ha.enabledtrueyarn.resourcemanager.ha.automatic-failover.enabledtrueyarn.resourcemanager.ha.automatic-failover.embeddedtrueyarn.resourcemanager.cluster-idyarn-clusteryarn.resourcemanager.ha.rm-idsrm1,rm2 !-这里RM主备结点需要单独指定,（可选） yarn.resourcemanager.ha.id rm2 -yarn.resourcemanager.scheduler.classorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairServal-ms5000yarn.resourcemanager.store.classorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStoreyarn.resourcemanager.zk-addresssht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181yarn.resourcemanager.zk.state-store.addresssht-sgmhadoopdn-01:2181,sht-sgmhadoopdn-02:2181,sht-sgmhadoopdn-03:2181yarn.resourcemanager.address.rm1sht-sgmhadoopnn-01:23140yarn.resourcemanager.address.rm2sht-sgmhadoopnn-02:23140yarn.resourcemanager.scheduler.address.rm1sht-sgmhadoopnn-01:23130yarn.resourcemanager.scheduler.address.rm2sht-sgmhadoopnn-02:23130yarn.resourcemanager.admin.address.rm1sht-sgmhadoopnn-01:23141yarn.resourcemanager.admin.address.rm2sht-sgmhadoopnn-02:23141yarn.resourcemanager.resource-tracker.address.rm1sht-sgmhadoopnn-01:23125yarn.resourcemanager.resource-tracker.address.rm2sht-sgmhadoopnn-02:23125yarn.resourcemanager.webapp.address.rm1sht-sgmhadoopnn-01:8088yarn.resourcemanager.webapp.address.rm2sht-sgmhadoopnn-02:8088yarn.resourcemanager.webapp.https.address.rm1sht-sgmhadoopnn-01:23189yarn.resourcemanager.webapp.https.address.rm2sht-sgmhadoopnn-02:231898.修改slavesrootsht-sgmhadoopnn-01 hadoop# vi slavessht-sgmhadoopdn-01sht-sgmhadoopdn-02sht-sgmhadoopdn-039.分发文件夹rootsht-sgmhadoopnn-01 hadoop# scp -r hadoop-2.7.2 rootsht-sgmhadoopnn-02:/hadooprootsht-sgmhadoopnn-01 hadoop# scp -r hadoop-2.7.2 rootsht-sgmhadoopdn-01:/hadooprootsht-sgmhadoopnn-01 hadoop# scp -r hadoop-2.7.2 rootsht-sgmhadoopdn-02:/hadooprootsht-sgmhadoopnn-01 hadoop# scp -r hadoop-2.7.2 rootsht-sgmhadoopdn-03:/hadoop八.启动集群另外一种启动方式:/bigdata/hadoop/hadoop2-cluster-ha-setup/1.启动zookeepercommand: ./zkServer.sh start|stop|statusrootsht-sgmhadoopdn-01 bin# ./zkServer.sh startJMX enabled by defaultUsing config: /hadoop/zookeeper/bin/./conf/zoo.cfgStarting zookeeper . STARTEDrootsht-sgmhadoopdn-01 bin# jps2073 QuorumPeerMain2106 Jpsrootsht-sgmhadoopdn-02 bin# ./zkServer.sh startJMX enabled by defaultUsing config: /hadoop/zookeeper/bin/./conf/zoo.cfgStarting zookeeper . STARTEDrootsht-sgmhadoopdn-02 bin# jps2073 QuorumPeerMain2106 Jpsrootsht-sgmhadoopdn-03 bin# ./zkServer.sh startJMX enabled by defaultUsing config: /hadoop/zookeeper/bin/./conf/zoo.cfgStarting zookeeper . STARTEDrootsht-sgmhadoopdn-03 bin# jps2073 QuorumPeerMain2106 Jps2.启动hadoop(HDFS+YARN)a.格式化前,先在journalnode 节点机器上先启动JournalNode进程rootsht-sgmhadoopdn-01 # cd /hadoop/hadoop-2.7.2/sbinrootsht-sgmhadoopdn-01 sbin# hadoop-daemon.sh start journalnodestarting journalnode, logging to /hadoop/hadoop-2.7.2/logs/.outrootsht-sgmhadoopdn-03 sbin# jps16722 JournalNode16775 Jps15519 QuorumPeerMainrootsht-sgmhadoopdn-02 # cd /hadoop/hadoop-2.7.2/sbinrootsht-sgmhadoopdn-02 sbin# hadoop-daemon.sh start journalnodestarting journalnode, logging to /hadoop/hadoop-2.7.2/logs/.outrootsht-sgmhadoopdn-03 sbin# jps16722 JournalNode16775 Jps15519 QuorumPeerMainrootsht-sgmhadoopdn-03 # cd /hadoop/hadoop-2.7.2/sbinrootsht-sgmhadoopdn-03 sb