CDH5.1.0hadoop-2.3.( 2+3 )0完全分布式集群配置与HA配置.doc

Hadoop-2.3.0-cdh5.1.0完全分布式集群配置及HA配置（冰峰）方式: 2个master + 3个slavehadoop-2.3.0-cdh5.1.0完全分布式集群配置HA配置重点说明:为了部署HA集群，应该准备以下事情：* NameNode服务器：运行NameNode的服务器应该有相同的硬件配置。* JournalNode服务器：运行的JournalNode进程非常轻量，可以部署在其他的服务器上。注意：必须允许至少3个节点。当然可以运行更多，但是必须是奇数个，如3、5、7、9个等等。当运行N个节点时，系统可以容忍至少(N-1)/2个节点失败而不影响正常运行。一、安装前准备：操作系统：CentOS6.564位操作系统环境：jdk1.7.0_45以上，本次采用jdk-7u72-linux-x64.tar.gzmaster151namenode节点master252namenode节点slave1 53datanode节点slave2 54datanode节点slave3:55datanode节点注：Hadoop2.0以上采用的是jdk环境是1.7,Linux自带的jdk卸载掉，重新安装下载地址：/technetwork/java/javase/downloads/index.html软件版本：hadoop-2.3.0-cdh5.1.0.tar.gz,zookeeper-3.4.5-cdh5.1.0.tar.gz下载地址：/cdh5/cdh/5/开始安装：二、jdk安装1、检查是否自带jdkrpm-qa|grepjdkjava-1.6.0-openjdk--1.1.el6.i6862、卸载自带jdkyum-yremovejava-1.6.0-openjdk--1.1.el6.i686安装jdk-7u72-linux-x64.tar.gz在usr/目录下创建文件夹java,在java文件夹下运行tarzxvfjdk-7u72-linux-x64.tar.gz解压到java目录下rootmaster01java#lsjdk1.7.0_72三、配置环境变量远行vi/etc/profile#/etc/profile#Systemwideenvironmentandstartupprograms,forloginsetup#Functionsandaliasesgoin/etc/bashrcexportJAVA_HOME=/usr/local/java/jdk1.7.0_65exportJRE_HOME=/usr/local/java/jdk1.7.0_65/jreexportCLASSPATH=/usr/local/java/jdk1.7.0_65/libexportPATH=$JAVA_HOME/bin:$PATH保存修改，运行source/etc/profile重新加载环境变量运行java-versionrootmaster01java#java-versionjavaversion1.7.0_72Java(TM)SERuntimeEnvironment(build1.7.0_72-b13)JavaHotSpot(TM)64-BitServerVM(build24.55-b03,mixedmode)Jdk配置成功四、系统配置预先准备5台机器，并配置IP,以下是我对这5台机器的角色分配。ip地址主机名NameNodeJournalNodeDataNodeZKHMsterHRegionServer51Master1是主是是是52Master2是从是53Slave1是是是是54Slave2是是是是55Slave3是是是journalnode共享文件集群,这个配置要注意,保证是奇数个子节点,具体在哪几台无关ZooKeeper 保证是奇数个子节点,具体在哪几台无关关闭防火墙chkconfigiptablesoff(永久性关闭)配置主机名和hosts文件rootmaster01java#vi/etc/hostslocalhostlocalhost.localdomainlocalhost4localhost4.localdomain4:1localhostlocalhost.localdomainlocalhost6localhost6.localdomain651 master152 master253 slave154 slave255 slave3按照不同机器IP配置不同的主机名SSH无密码验证配置 ssh-keygen -t rsa /每个节点都要执行cp id_rsa.pub authorized_keys /主节点执行就行.之后直接scp到其他节点即可 4.ss-copy-id -i master1 /在master2 slave1 slave2 slave3节点上都执行一次或者执行:在master1上执行:scp -r /.ssh/authorized_keys master2:/.ssh/ scp -r /.ssh/authorized_keys slave1:/.ssh/ scp -r /.ssh/authorized_keys slave2:/.ssh/ scp -r /.ssh/authorized_keys slave3:/.ssh/ rootslave01#免密码登陆不同节点,手动登陆时已被写入免密码登陆Master1-master1 Master1-slave1 Master1-slave2 Master1-slave3Master2-master2 Master2-slave1 Master2-slave2 Master2-slave3执行相同的操作。五、安装Hadoop建立文件目录/usr/local/cloudcloud下创建文件夹:mkdir data 存放数据Data下mkdir logs日志文件解压tar zxvf haooop原文件, mv hadoop* hadoop解压tar zxvf zookeeper原文件 mv zookeeper* zookeeper-注意环境变量的配置-export JAVA_HOME=/usr/local/java/jdk1.7.0_72export JRE_HOME=/usr/local/java/jdk1.7.0_72/jreexport ZOOKEEPER_HOME=/usr/local/cloud/zookeeperexport HADOOP_HOME=/usr/local/cloud/hadoopexport CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/lib/*.jar:$JAVA_HOME/jre/lib/*.jarExport PATH=.:$JAVA_HOME/bin:$JRE_HOME/bin:$ZOOKEEPER_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH-rootslave01cloud#lsdatahadoopzookeeper5.1、配置hadoop-env.sh进入到/usr/local/cloud/hadoop/etc/hadoop目录下配置vihadoop-env.shhadoop运行环境加载exportJAVA_HOME=/usr/local/java/jdk1.7.0_72在末尾添加两行:export HADOOP_COMMON_LIB_NATIVE_DIR=/usr/local/cloud/hadoop/lib/nativeexport HADOOP_OPTS=-Djava.library.path=/usr/local/cloud/hadoop/lib注意此两行是为了解决如下情况的:浏览器输入 http:/master1:9000/显示如下的一段文字，ItlookslikeyouaremakinganHTTPrequesttoaHadoopIPCport.Thisisnotthecorrectportforthewebinterfaceonthisdaemon.5.2、配置core-site.xmlfs.defaultFShdfs:/ # 如果想上传数据就启用hdfs存文件配置,才可以hdfs:/master1:9000hadoop.tmp.dir/usr/local/cloud/data/hadoop/xyuser.root.groups*ha.zookeeper.quorum 这里要和hdfs-site.xml里保持一致master1:2181,slave1:2181,slave2:2181(2)hdfs-site.xml配置.dir/usr/local/cloud/data/hadoop/dfs/nndfs.datanode.data.dir/usr/local/cloud/data/hadoop/dfs/dndfs.support.append #做优化使用的配置部分nodes.nngnn1,node.rpc-address.nng.nn1master1:9000node.rpc-address.nng.nn2master2:9000node.http-address.nng.nn1master1:50070node.http-address.nng.nn2master2:50070node.servicerpc-address.nng.nn1master1:53310node.servicerpc-address.nng.nn2master2:53310node.shared.edits.dir qjournal:/master1:8485;slave1:8485;slave2:8485/nngdfs.journalnode.edits.dir/usr/local/cloud/data/hadoop/ha/node.ha.ConfiguredFailoverProxyProviderdfs.ha.automatic-failover.enabledtruedfs.ha.fencing.methodssshfencedfs.ha.fencing.ssh.private-key-files/root/.ssh/id_rsaha.zookeeper.quorum 这里要和core-site.xml里保持一致master1:2181,slave1:2181,slave2:2181 # 先不配置ha.zookeeper.session-timeout.ms 10000 # 先不配置dfs.ha.fencing.ssh.connect-timeout 10000 以毫秒为单位,默认30000 5.3配置yarn # 先不配置mapreduce.jobhistory.address master:10020 # 先不配置mapreduce.jobhistory.webapp.address master:19888 5.4配置编辑文件$HADOOP_HOME/etc/hadoop/slaves，修改内容如下所示slave1slave2slave3表示以上三个节点作为DataNode节点。5.5配置yarnHA配置yarn-env.shjava环境#someJavaparametersexportJAVA_HOME=/usr/local/java/jdk1.7.0_72 5.6配置yarn-site.xmlyarn.resourcemanager.connect.retry-interval.ms2000yarn.resourcemanager.ha.enabledtrueyarn.resourcemanager.ha.automatic-failover.enabledtrueyarn.resourcemanager.cluster-idnngyarn.resourcemanager.ha.rm-idsnn1,nn2 yarn.resourcemanager.hostname.nn1 master1 yarn.resourcemanager.hostname.nn2 master2 yarn.resourcemanager.recovery.enabledtrueyarn.resourcemanager.zk-state-store.addressmaster1:2181注意这里配置,localhost:2181这个也是可以的yarn.resourcemanager.store.classorg.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStoreyarn.resourcemanager.zk-addressmaster1:2181,slave1:2181,slave2:2181 注意: 这里置erval-ms5000yarn.resourcemanager.address.rm1master1:23140yarn.resourcemanager.address.nn1master1:23140yarn.resourcemanager.scheduler.address.nn1master1:23130yarn.resourcemanager.webapp.address.nn1master1:23188yarn.resourcemanager.resource-tracker.address.nn1master1:23125yarn.resourcemanager.admin.address.nn1master1:23141yarn.resourcemanager.ha.admin.address.nn1master1:23142yarn.resourcemanager.address.nn2master2:23140yarn.resourcemanager.scheduler.address.nn2master2:23130yarn.resourcemanager.webapp.address.rm2master2:23188yarn.resourcemanager.resource-tracker.address.nn2master2:23125yarn.resourcemanager.admin.address.nn2master2:23141yarn.resourcemanager.ha.admin.address.nn2master2:23142yarn.nodemanager.localizer.addressslave1:23344,slave2:23344,slave3:23344yarn.nodemanager.webapp.addressmaster1:23999 yarn.nodemanager.aux-servicesmapreduce_shuffleyarn.nodemanager.aux-services.mapreduce.shuffle.classorg.apache.hadoop.mapred.ShuffleHandleryarn.nodemanager.local-dirs/home/hadoop/data/yarn/localyarn.nodemanager.log-dirs/home/hadoop/data/yarn/logsmapreduce.shuffle.port23080yarn.client.failover-proxy-providerorg.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider至此: 要手动创建的文件夹有:/usr/local/cloud/下:/usr/local/cloud/data/hadoop/dfs/dn/usr/local/cloud/data/hadoop/dfs/nn/usr/local/cloud/data/hadoop/ha/journal/usr/local/cloud/data/hadoop/logs/hadoop/usr/local/cloud/data/hadoop/tmp/usr/local/cloud/data/hadoop/yarn/local/usr/local/cloud/data/hadoop/yarn/logs配置zookeeper集群 详细说明参考: /hbase/book.html#zookeeper在zookeeper目录下建立data目录和logs目录，zookeeper可以在Hadoop集群中选几台机器同时作为zookeeper节点，给HA私用。在实践中建议采用手工切换的方式，这样更可靠，也方便查找问题。进入zookeeper目录conf下,mv cp zoo_sample.cfg zoo.cfg 之后配置vim zoo.cfg#手动创建数据存放目录数据存放目录dataDir=/usr/local/cloud/zookeeper/data #日志存放目录，手动创建。如果没有貌似启动不成功dataLogDir=/usr/local/cloud/zookeeper/logs#theportatwhichtheclientswillconnectclientPort=2181注意:zookeeper可以在Hadoop集群中选几台机器同时作为zookeeper节点，给HA私用。在实践中建议采用手工切换的方式，这样更可靠，也方便查找问题 server.1=master1:2888:3888server.2=master2:2888:3888server.3=slave1:2888:3888server.4=slave2:2888:3888server.5=slave3:2888:3888在data目录下创建myid文件，并在对应的机器上填写数字，cd /usr/local/cloud/zookeeper/data/myid如上配置拷贝到不同节点里,命令:scp -r /usr/local/cloud master2:/usr/local/master1server.1中的data的myid写入1，master2server.2中的data的myid写入2，slave1server.3中的data的myid写入3，slave2server.4 中的data的myid写入4，slave3server.5 中的data的myid写入5，在各个机器下zookeeper目录下的bin目录下执行cd /usr/local/cloud/zookeeper/bin/命令: zkServer.sh start再运行: zkServer.sh status 如果出现leader或fllower则说明集群配置正确。到此各个配置文件配置完毕启动Hadoop集群严格按照以下顺序执行 注意: 执行命令启动集群,以下命令严格注意执行顺序，不能颠倒！启动zookeeper集群的节点master1,master2,slave1,slave2,slave3执行:/usr/local/cloud/zookeeper/bin/zkServer.sh start/usr/local/cloud/zookeeper/bin/zkServer.sh status(2) 启动JournalNode集群的节点，在master1，slave1,slave2/usr/local/cloud/hadoop/sbin/hadoop-daemon.sh start journalnode(3)在 master1,master2上cd /usr/local/cloud/hadoop/bin/ 下hdfs zkfc -formatZK进行格式化创建命名空间（4.1）在主节点master1上执行格式化一个NameNode/usr/local/cloud/hadoop/bin/hadoop namenode -format nng（4.2）在主节点master1上执行启动namenode主机/usr/local/cloud/hadoop/sbin/hadoop-daemon.sh start namenode(5.1)在备主节点master2上执行格式化另一个NameNode/usr/local/cloud/hadoop/bin/hdfs namenode -bootstrapStandby(5.2)在备主节点master2上执行启动namenode备机/usr/local/cloud/hadoop/sbin/hadoop-daemon.sh start namenode(6)在master1和master2两个namenode节点都启动执行以下命令/usr/local/cloud/hadoop/sbin/hadoop-daemon.sh start zkfc(7)启动DataNodes 在slave1,slave2,slave3节点启动datanode都执行以下命令/usr/local/cloud/hadoop/sbin/hadoop-daemon.sh start datanode或者如下: 在master1或者在master2上执行命令: 会分别启动3个节点的DataNode。/usr/local/cloud/hadoop/sbin/hadoop-daemons.sh start datanode 这时候HA集群就启动了.(8) 在主master1主节点启动namenode的yarn框架， 运行start-yarn.sh命令/usr/local/cloud/hadoop/sbin/start-yarn.sh start yarn注意:如果是第二次以上执行格式化前就删除以下目录下文件,再次格式化操作/usr/local/cloud/data/hadoop/dfs/dn/current/usr/local/cloud/data/hadoop/dfs/nn/current/usr/local/cloud/data/hadoop/ha/journal/usr/local/cloud/data/hadoop/tmp/usr/local/cloud/data/hadoop/yarn/local/执行的日志文件:/usr/local/cloud/hadoop/logs/usr/local/cloud/data/logs/hadoopjps可以看到namenode节点rootmaster1 # jps3960 NameNode3705 QuorumPeerMain4176 ResourceManager HA在namenode的进程4070 DFSZKFailoverController3839 JournalNode 和namenode通信启用的4246 Jpsrootmaster2 # jps3407 NameNode3558 Jps3512 DFSZKFailoverController3272 QuorumPeerMainrootslave1 # jps3543 DataNode3623 Jps3471 JournalNode3392 QuorumPeerMainrootslave2 # jps3128 QuorumPeerMain3207 JournalNode3279 DataNode3349 Jpsrootslave3 # jps3281 Jps3211 DataNode3139 QuorumPeerMain这时候，使用浏览器访问 http:/master1:50070 和 http:/master2:50070 如果能够看到两个页面，证明NameNode启动成功了。这时，两个NameNode的状态都是standby。实验一下NameNode切换,转换active ,先kill -9 端口号, 干掉NN切换NameNode的运行状态在HA模式下，NameNode启动后会直接进入standby模式，可通过以下命令将NameNode切换为active模式测试通过两种方式模拟Namenode异常终止NameNode进程直接重启active NN两种方式NameNode状态均切换成功，且保持与client及Datanode端的正常通信。不同之处在于对于直接kill active NameNode，会有zookeeper主动释放锁，而掉电重启active NN所运行的服务器则需等待zookeeper锁超时方能进行Namenode状态切换，由此在掉电的情景中状态切换的过程更长查询NameNode状态 执行命令 : 查询nn1的NameNode状态:hdfs haadmin -getServiceState nn1在指定的name service下的手动切换NameNode的状态:hdfs haadmin -ns name service -failover standby active如: hdfs haadmin -ns nng -failover nn1 nn2再使用浏览器访问 http:/master1:50070 和 http:/master2:50070，会发现master1的nn1节点变为standby，master2的nn2变成了active。如果向上传数据，还需要修改core-site.xml中的的值，改为hdfs:/master1:9000 才行,用浏览器访问 http:/master1:9000-hbase0.98.1的安装下载解压# 解压Hbase-0.98.1-cdh5.1.0.tar.gz（启动双HMaster的配置，master1是主HMaster，master2是从HMaster）、slave1、slave2、slave3（3台HRegionServer）安装路径如下：/usr/local/cloud/hbase/加入环境变量 vim /etc/pr