CentOS 6.5安装19实体节点Hadoop2.2.0实施过程.docx

CentOS 6.5 x64下安装19个节点(2个NameNode+Yarn，1个JournalNode,16个12块4TB硬盘的数据节点)Hadoop 2.2.0集群配置指南一、 准备工作1. 操作系统光盘：CentOS Linux 6.5 x642. hadoop 2.2.0软件安装包：hadoop-2.2.0.tar.gz ， 下载地址：/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz3. 节点硬件：a) 主控节点3台：Dell PowerEdge R620,Xeon E5-2650v2*2，128G RAM，300GB*4(RAID5)b) 数据节点16台：Dell PowerEdge R720xd，Xeon E5-2609v2*2，64G RAM，4TB*12(RAID设置成12个以单独1个物理硬盘组成的RAID0虚拟盘)二、 节点规划1. IP地址与节点主机名规划：a) NameNode+Yarn节点1：IP：，主机名：hd-nn1b) NameNode+Yarn节点2：IP：，主机名：hd-nn2c) JournalNode节点：IP：，主机名：hd-jnd) 数据节点1：IP：，主机名：hd-datanode1e) 数据节点2：IP：，主机名：hd-datanode2f) 数据节点3：IP：，主机名：hd-datanode3g) 数据节点4：IP：，主机名：hd-datanode4h) 数据节点5：IP：，主机名：hd-datanode5i) 数据节点6：IP：0，主机名：hd-datanode6j) 数据节点7：IP：1，主机名：hd-datanode7k) 数据节点8：IP：2，主机名：hd-datanode8l) 数据节点9：IP：3，主机名：hd-datanode9m) 数据节点10：IP：4，主机名：hd-datanode10n) 数据节点11：IP：5，主机名：hd-datanode11o) 数据节点12：IP：6，主机名：hd-datanode12p) 数据节点13：IP：7，主机名：hd-datanode13q) 数据节点14：IP：8，主机名：hd-datanode14r) 数据节点15：IP：9，主机名：hd-datanode15s) 数据节点16：IP：0，主机名：hd-datanode162. 数据节点分区规划a) 数据节点系统分区：/dev/sda2500Mext4/boot/dev/sda1200MEFI/boot/efi/dev/sda3300Gext4/dev/sda4100Gext4/home/dev/sda516Gswapb) 数据节点的数据分区：/dev/sda63.2Text3/data1/dev/sdb13.6Text3/data2/dev/sdc13.6Text3/data3/dev/sdd13.6Text3/data4/dev/sde13.6Text3/data5/dev/sdf13.6Text3/data6/dev/sdg13.6Text3/data7/dev/sdh13.6Text3/data8/dev/sdi13.6Text3/data9/dev/sdj13.6Text3/data10/dev/sdk13.6Text3/data11/dev/sdl13.6Text3/data12c) NameNode节点和Yarn节点分区要求Swap分区设定为16G（16384MB）三、 操作系统安装a) 语言选择：基础语言为en_US，时区为Shanghai/Asiab) 安装包选择：选择Basic Server,选择Customize Now,选择Language中Chinese Support。四、 多网卡绑定设置考虑到hadoop内部各节点之间网络传输的流量巨大，而且需要可靠性，因此将每台服务器的4块网卡进行LACP捆绑聚合，实现网卡的负载均衡与高可用性，让服务器的网络带宽从理论上扩展3-4倍。以下步骤在每一个节点上以root用户执行：a) 编辑/etc/modprobe.d/bonding.conf文件#vi /etc/modprobe.d/bonding.confalias bond0 bondingoptions bond0 miimon=100 mode=4b) 加载bonding模块#modprobe bondingc) 编辑/etc/sysconfig/netwoork文件#vi /etc/sysconfig/networkNETWORKING=yesHOSTNAME=hd-nn1(修改成本机的主机名)GATEWAY=54d) 编辑/etc/sysconfig/network-script/ifcfg-bond0文件#vi /etc/sysconfig/network-script/ifcfg-bond0DEVICE=bond0TYPE=EthernetONBOOT=yesBOOTPROTO=noneIPADDR=(修改成本机的IP)PREFIX=24NETMASK=NETWORK=BOARDCAST=55GATEWAY=54DNS1=14DEFROUTE=yesIPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noNAME=bond0e) 编辑修改/etc/sysconfig/network-script/ifcfg-em1文件#vi /etc/sysconfig/network-script/ifcfg-em1DEVICE=em1TYPE=EthernetUUID=1a63f70f-c143-4455-b7f7-857cae837d09 (保留系统默认值)ONBOOT=yesBOOTPROTO=noneHWADDR=C8:1F:66:E4:18:E2 (保留系统默认值)IPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond0SLAVE=yesNAME=System em1f) 编辑修改/etc/sysconfig/network-script/ifcfg-em2文件#vi /etc/sysconfig/network-script/ifcfg-em2DEVICE=em2TYPE=EthernetUUID= 609c41e3-3db0-43b0-89f0-cf7a7b896070 (保留系统默认值)ONBOOT=yesBOOTPROTO=noneHWADDR= C8:1F:66:E4:18:E3 (保留系统默认值)IPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond0SLAVE=yesNAME=System em2g) 编辑修改/etc/sysconfig/network-script/ifcfg-em3文件#vi /etc/sysconfig/network-script/ifcfg-em3DEVICE=em3TYPE=EthernetUUID= 66819fc0-65af-428a-95ff-da06be34c896 (保留系统默认值)ONBOOT=yesBOOTPROTO=noneHWADDR= C8:1F:66:E4:18:E4 (保留系统默认值)IPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond0SLAVE=yesNAME=System em3h) 编辑修改/etc/sysconfig/network-script/ifcfg-em4文件#vi /etc/sysconfig/network-script/ifcfg-em4DEVICE=em4TYPE=EthernetUUID= 5b2e7f76-4ba8-418a-9f14-c867786f65d5 (保留系统默认值)ONBOOT=yesBOOTPROTO=noneHWADDR= C8:1F:66:E4:18:E5 (保留系统默认值)IPV4_FAILURE_FATAL=yesIPV6INIT=noUSERCTL=noMASTER=bond0SLAVE=yesNAME=System em4i) 修改完成后，重启服务器，使之生效。重启完成后可以通过ping 54和ifconfig -a命令查看捆绑是否成功，也可以通过查看cat /proc/net/bonding/bond0查看捆绑每个端口连接情况：#cat /proc/net/bonding/bond0Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)Bonding Mode: IEEE 802.3ad Dynamic link aggregationTransmit Hash Policy: layer2 (0)MII Status: upMII Polling Interval (ms): 100Up Delay (ms): 0Down Delay (ms): 0802.3ad infoLACP rate: slowAggregator selection policy (ad_select): stableActive Aggregator Info:Aggregator ID: 1Number of ports: 4Actor Key: 17Partner Key: 18Partner Mac Address: 14:14:4b:76:46:4cSlave Interface: em1MII Status: upSpeed: 1000 MbpsDuplex: fullLink Failure Count: 0Permanent HW addr: c8:1f:66:e4:18:e2Aggregator ID: 1Slave queue ID: 0Slave Interface: em2MII Status: upSpeed: 1000 MbpsDuplex: fullLink Failure Count: 0Permanent HW addr: c8:1f:66:e4:18:e3Aggregator ID: 1Slave queue ID: 0Slave Interface: em3MII Status: upSpeed: 1000 MbpsDuplex: fullLink Failure Count: 0Permanent HW addr: c8:1f:66:e4:18:e4Aggregator ID: 1Slave queue ID: 0Slave Interface: em4MII Status: upSpeed: 1000 MbpsDuplex: fullLink Failure Count: 0Permanent HW addr: c8:1f:66:e4:18:e5Aggregator ID: 1Slave queue ID: 0五、 Hadoop部署安装在每台服务器上执行如下指令：a) 创建hadoop用户在每个节点上以root用户登录，创建hadoop用户：#useradd hadoop#passwd hadoop每个节点的hadoop用户密码相同。暂时设置为nebula。b) 创建/opt/hadoop目录并修改权限属性在每个节点上，以root用户登录后执行#mkdir /opt/hadoop在所有节点上以root权限执行：#chown R hadoop:hadoop /opt/hadoop#chmod R 755 /opt/hadoopc) 修改数据目录权限属性在每个数据节点上，以root用户登录后执行：#chown -R hadoop:hadoop /data1#chown -R hadoop:hadoop /data2#chown -R hadoop:hadoop /data3#chown -R hadoop:hadoop /data4#chown -R hadoop:hadoop /data5#chown -R hadoop:hadoop /data6#chown -R hadoop:hadoop /data7#chown -R hadoop:hadoop /data8#chown -R hadoop:hadoop /data9#chown -R hadoop:hadoop /data10#chown -R hadoop:hadoop /data11#chown -R hadoop:hadoop /data12d) 关闭防火墙在所有节点上以root权限执行：#service iptables stop#chkconfig iptables offe) 修改/etc/hosts在hd-nn1节点上，以root用户执行 #vi /etc/hosts ，添加以下内容：hd-nnhd-nnhd-jnhd-datanodehd-datanodehd-datanode3hd-datanode4hd-datanode50hd-datanode61hd-datanode72hd-datanode83hd-datanode94hd-datanode105hd-datanode16hd-datanode17hd-datanode138hd-datanode149hd-datanode150hd-datanode16然后通过scp命令复制到每台节点上：#scp /etc/hosts roothd-nn2:/etc/#scp /etc/hosts roothd-jn:/etc/#scp /etc/hosts roothd-datanode1:/etc/#scp /etc/hosts roothd-datanode2:/etc/#scp /etc/hosts roothd-datanode3:/etc/#scp /etc/hosts roothd-datanode4:/etc/#scp /etc/hosts roothd-datanode5:/etc/#scp /etc/hosts roothd-datanode6:/etc/#scp /etc/hosts roothd-datanode7:/etc/#scp /etc/hosts roothd-datanode8:/etc/#scp /etc/hosts roothd-datanode9:/etc/#scp /etc/hosts roothd-datanode10:/etc/#scp /etc/hosts roothd-datanode11:/etc/#scp /etc/hosts roothd-datanode12:/etc/#scp /etc/hosts roothd-datanode13:/etc/#scp /etc/hosts roothd-datanode14:/etc/#scp /etc/hosts roothd-datanode15:/etc/#scp /etc/hosts roothd-datanode16:/etc/然后在每个节点上用ping 主机名，测试每个主机名是否正确识别并连通。f) 设置各节点无密码ssh登录1) 以hadoop用户登录所有节点，以下所有操作均以hadoop用户执行2) 在每个节点上生成RSA密钥对：$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): /使用默认位置，直接回车 Enter passphrase (empty for no passphrase): /直接回车 Enter same passphrase again: Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: 5b:47:52:a7:1e:1c:4d:98:d7:04:eb:d9:0e:5f:dd:3d hadoophadoopThe keys randomart image is: +- RSA 2048-+ | o=+.| | o+o.| | . =. | | + o o+| | S . o +E*| | o . +o| | o| | | | +-+ 3) 在hd-nn1节点上以hadoop用户执行：$cp /.ssh/id_rsa.pub /.ssh/authorized_keys $chmod 644 /.ssh/authorized_keys$scp hadoophd-nn2:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-jn:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode1:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode2:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode3:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode4:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode5:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode6:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode7:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode8:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode9:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode10:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode11:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode12:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode13:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode14:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode15:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp hadoophd-datanode16:/.ssh/id_rsa.pub ./$cat id_rsa.pub /.ssh/authorized_keys$scp /.ssh/authorized_keys hadoophd-nn2:/.ssh/$scp /.ssh/authorized_keys hadoophd-jn:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode1:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode2:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode3:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode4:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode5:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode6:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode7:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode8:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode9:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode10:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode11:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode12:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode13:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode14:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode15:/.ssh/$scp /.ssh/authorized_keys hadoophd-datanode16:/.ssh/4) 在所有节点上以hadoop用户执行以下操作：$chmod 600 /.ssh/authorized_keys$ssh hd-nn1输入yes$exit$ssh hd-nn1此时不需要输入任何内容与密码直接登录成功方为正确。$exit$ssh hd-nn2输入yes$exit$ssh hd-nn2此时不需要输入任何内容与密码直接登录成功方为正确。$exit$ssh hd-jn输入yes$exit$ssh hd-jn此时不需要输入任何内容与密码直接登录成功方为正确。$exit$ssh hd-datanode1输入yes$exit$ssh hd-datanode1此时不需要输入任何内容与密码直接登录成功方为正确。$exit$ssh hd-datanode2输入yes$exit$ssh hd-datanode2此时不需要输入任何内容与密码直接登录成功方为正确。$exit以同样的方式，一直执行到hd-datanode16，然后在每一台节点上都完全一样地完整执行一遍。5) 所有节点免密码ssh配置成功。g) 配置所有节点的/etc/profile在hd-nn1节点上以root用户操作：#vi /etc/profile在文件末尾添加：#hadoop variable settingsexport HADOOP_HOME=/opt/hadoop/hadoop-2.2.0export HADOOP_PREFIX=$HADOOP_HOMEexport HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport HADOOP_MAPRED_HOME=$HADOOP_HOMEexport HADOOP_YARN_HOME=$HADOOP_HOMEexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport MAVEN_HOME=/opt/hadoop/maven export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib:$MAVEN_HOME/bin# Native Pathexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_PREFIX/lib/nativeexport HADOOP_OPTS=-Djava.library.path=$HADOOP_PREFIX/lib/native保存退出后执行：#soure /etc/profile然后用scp复制到所有节点上：#scp /etc/profile hd-nn2:/etc#scp /etc/profile hd-jn:/etc#scp /etc/profile hd-datanode1:/etc#scp /etc/profile hd-datanode2:/etc#scp /etc/profile hd-datanode3:/etc#scp /etc/profile hd-datanode4:/etc#scp /etc/profile hd-datanode5:/etc#scp /etc/profile hd-datanode6:/etc#scp /etc/profile hd-datanode7:/etc#scp /etc/profile hd-datanode8:/etc#scp /etc/profile hd-datanode9:/etc#scp /etc/profile hd-datanode10:/etc#scp /etc/profile hd-datanode11:/etc#scp /etc/profile hd-datanode12:/etc#scp /etc/profile hd-datanode13:/etc#scp /etc/profile hd-datanode14:/etc#scp /etc/profile hd-datanode15:/etc#scp /etc/profile hd-datanode16:/etch) 下载并安装最新版JDK7u71从oracle官网上下载jdk-7u71-linux-x64.tar.gz，拷贝到所有节点的/opt/hadoop目录下。下载地址：/technetwork/java/javase/downloads/jdk7-downloads-1880260.html在所有节点上，以hadoop用户解压安装jdk：$cd /opt/hadoop$tar zxvf jdk-7u71-linux-x64.tar.gz然后配置hadoop用户的JAVA_HOME环境变量：$vi /.bash_profile在末尾添加：export JAVA_HOME=/opt/hadoop/jdk1.7.0_71保存退出后执行：$ . /.bash_profilei) 下载并解压hadoop-2.2.0-src由于官方版的hadoop-2.2.0是以32位编译的，而CentOS 6.5是64位的系统，因此需要重新以64位编译hadoop-2.2.0-src在hd-nn1上以hadoop用户执行：$cd /opt/hadoop$wget /apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz下载完成后执行：$tar zxvf hadoop-2.2.0-src.tar.gzj) 编译hadoop-2.2.0(1)下载并安装maven以hadoop用户登录$cd /opt/hadoop$wget /apache/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.zip$unzip apache-maven-3.1.1-bin.zip$mv apache-maven-3.1.1 maven(2)修改/opt/hadoop/maven/conf/settings.xml文件，在里添加，原本的不要动 nexus-osc * Nexusosc /content/groups/public/ 同样，在内新添加 jdk-1.7 1.7 nexus local private nexus /content/groups/public/ true false nexus local private nexus /content/groups/public/ true false (3)安装编译所需环境以root用户执行：$su# yum install -y gcc gcc-c+ make cmake openssl-devel ncurses-devel(4)下载ptotoc并解压# wget /files/protobuf-2.5.0.tar.gz#tar zxvf /files/protobuf-2.5.0.tar.gz(5)编译并安装protoc# cd protobuf-2.5.0# ./configure# make# make check# make install(6)编译hadoop clean#exit$cd /opt/hadoop/hadoop-2.2.0-src$ mvn clean install DskipTests出现如下画面时则为成功(7)编译hadoop$ mvn package -Pdist,native -DskipTests Dtar编译成功画面：将编译好的hadoop移动到/opt/hadoop目录下$mv /opt/hadoop/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0 /opt/hadoop查看hadoop版本是否编译成功：$ hadoop versionHadoop 2.2.0Subversion Unknown -r UnknownCompiled by hadoop on 2014-12-24T12:59ZCompiled with protoc 2.5.0From source with checksum 79e53ce7994d1628b240f09af91e1af4This command