Hadoop集群部署方案设计

1、实用文档Hadoop集群部署方案目录1. 网络拓扑12. 软件安装12.1. 修改主机名122修改host文件12.3. 创建Hadoop用户224禁用防火墙22.5. 设置ssh登录免密码22.6. 安装 hadoop43. 集群配置53.1. 修改脚本53.1.1. hadoop-env.sh. yarn-env.sh5.3.2. 配置文件53.2.1. core-site.xml. hdfs-site.xml. mapred-site.xml103.2.4. yarn-site.xml.. 酉己置 data node133.3.

2、创建目录 144. 启动zk集群145启动 hadoop 145.1. 启动所有节点 journalnode145.2. 格式化 h1 namenode145.3. 在h1上格式化ZK.155.4. 启动 h1 的 n ame no de，zkfc165.5. 启动 h2 上 namenode1656 同步hl上的格式化数据到 h2165.7. 启动 HDFS175.8. 启动 YARN185.9. 启动 h2 ResourceManager185.10. h4 上启动 JobHistoryServer195.11. 查看 ResourceManager状态196. 浏览器访问196.1. n

3、amenode 管理界面 196.1.1. 67:50070 196.1.2. 68:50070 206.2. ResourceManager管理界面206.2.1. 67:8088/216.2.2. 68:8088/216.3. JournalNode HTTP服务216.3.1. 67:8480/216.4. Data node HTTP服务226.4.1. 67:

4、50075/226.5. jobhistory 管理界面226.5.1. 70:19888/jobhistory227. 测试237.1. 上传文件至hdfs237.2. 验证 hdfs HA237.3. 验证 yarn241. 网络拓扑h1h2h3h4IP66786970rolematsterslaverslaverslaverName node是（active）是（sta ndby）否否Data node是是是是ResourceMa nager是是否

5、否NodeMa nager是是是是zkfc是是否否Jour nalnode是是是否JobHistory否否否是2. 软件安装2.1.修改主机名修改四台机器的主机名，hl, h2, h3, h4查看是否修改成功un ame -a22修改host文件修改三台机器/etc/hosts文件配置hadoop节点k92.163.12L.L7 hl68 h269 h70 h4配置zk节点3 no del4 node25 fiode32.3.创建Hadoop用

6、户1）创建特定的Hadoop用户帐号以区分Hadoop和本机上的其他服务sudo groupadd hadoopuseradd hadooppasswd hadoopusermod -a -G hadoop hadoop2) 给hadoop用户添加权限，打开/etc/sudoers文件:在root ALL=(ALL) ALL后面添加hadoop ALL=(ALL) ALL2.4. 禁用防火墙chkc onfig iptables off2.5. 设置ssh登录免密码Hadoop控制脚本（并非守护进程）依赖SSH来执行针对整个集群的操作。例如，Name node是通过SSH（ Secure Sh

7、ell）来启动和停止各个 data node上的各种守护进程的为了支持无缝式工作，SSH安装好之后，需要允许hadoop用户无需键入密码即可登陆集群内的机器。最简单的方法是每个机器创建一个无密码公钥/私钥对1）生成密钥胃？?31 抽!、EF Pt/t 口/h&me/hadoflp/.ssh/id_rsa. public key has been saved in /hoime/hadocp/.HEh/id_rsa.pub .了7广證网叱貪霽遴蹲幫誓綴 “ z十 /.ssh/authorized_keys” 命令，将这几个节点上的共钥 id_ras.pub拷贝到 h1 中的 authorized

8、_keys文件中hadooph? .sshlf cat */.ssh/id rsa.pub | ssh hadoophl cat-/.ssh/authorized keysiddoophls passward：Rd f 強 山MAB测萄怦?F AAAhB Alh -J59nS J i-*PjI 吨眄rttV jiT弧UlVSumHLP蕾酣 帥命勺 油归 0G2i气卩已班 ioYOHKHTcM|JpSH 判 如 口1 毎审声 Q|ifi77lJfnaMrOf37l 血箕 TYEfbAFSU p 讪 vCLJkfLCWtj SjKCMF/ieI 刖0|：加仙羽鼻宿:沏艸 伽 腔 眄血师 j 1L1

9、|拷贝完成后，hl中的authorized_keys文件内容Sgzp&l 矶应和tMUSfiYrtj乩炖恥 LL4y/f g 4fnPLKJlTlUC 3c L /RHHMriMKlOKLE 伯 t ：f 厂 We j43/8_HKtH_Z5 z ig 到打生畑件“如离匚肿 q xZc Jbtisk-VHtK hdd0Q%hlw rsa AWS63ikCLyc2A46AAMQEdWdVlPt価辎舁如讥Z/iwGISME / 3YZW/sIC?E I y+SCha*f代3呻讪號评咽硼上乩*叩说沖丿缶归FF$妙SQ咖訥QH吐可訓EWgTCGl v/q2t BpwGJVpbrzk（K733w IvA

10、X2M54Z1J4C/VKT rVatq4HKpCWt jS?2d 呷j洲帀旳eXRwFS7s*ZTTF 丿 upQ旳 伽血西 T quwtpirijfwHji / ijod j 晦和32 初电 珂 kl*9百w出Ei缈jgcodu瞄 K吃l“细athdQ比吕碍 hadgp#i】ssi-rsa MJ抽沁蠡切2LilwiXAx2tp*5KyXTll-jDIrPCc丈点J匕丫Jq2n去穴dMu ajg廿丄Lc咋 FLrtigJVdDXrlcvl&lwjulill OqLw和血“甜存曾B4785n擁“钢黄GL升b（时vkOJiead轲心iflq/hg撇?TBl询Hon直fq理lICTi和HkFWMH

11、即勵和仇dCFLTME肝0剜 CE-Tui1nU5565x7iaji r31tfcha7qB5CH3S-l7irj ahltCZ-wr&Ofc 也阳aiTaFbolFBSCEd Dj Oh IWi j LtvWtuiragWMgitlUQe： fn 町 q L/kLj5c：fllWtflH hddooph27）将hl中的authorized_keys文件分发到其他节点（h2、h3、h4）上，在hadoopl 上，使用 scp -r /.ssh/authorized_keys hadooph1:/.ssh/ 命令分发8）测试hadoopl ,sshl$ ssh h2Last login： Tue

12、 Mar 31 17:43:32 2017 from h2 h3dooph2 7$ exitlogoutConnection to h2 closed*26 安装 hadoop计算md5值，看文件是否完整hadoophl local? cat ./hadoop-2.7.3.tar gz.mds Igrep MD亍 doop-2.7,3.tar.z:MD5 - 34 55 BB 57 E4 B4 90 6B BE A6 7B 53 CC A7 8F A8hadoopfihl local$ mdSsurn ./hadaop-2k7.3tar.gz |tr a-z A-Z3455BB57E 484

13、9G6BBE A67B5SCC A78F A8 VHADODP-2.7.3. TAR. &Z#解压到/usr/local中#将文件夹名改为hadoopsudo tar -zxvf ./hadoop-2.7.3.tar.gzsudo mv ./hadoop-2.7.3/ ./hadoopsudo chow n -R hadoop ./hadoophadoophs binlf ./hadoop versionHadoap 2.7.3Subversion https；/git-wip us.apache,org/repos/asf/hadoop.git -r baa91f7c6bc9cb92be59

14、&2de4719clc8af9Lccff Cotnpilod by root on 2015 09 1ET01:1ZConipiled with protoc 2.5BFrom source with checksjm 2e4ce5f957ea4db193bce3734ff29ff*This command 丽s run using /usr/l&caVhadQOp/share/hadQQp/coniiKjn/hadooi）Conon-2f7H3tjar为了方便，使用hadoop命令或者start-all.sh等命令，修改Master上/etc/profile 新export HADOOP_H

15、OME=/usr/local/hadoopexport PATH=$PATH:$HADOOP_HOME/bin source /etc/profile 使生效3. 集群配置3.1.修改脚本311 hadoop-env .sh$vi hadoop-e nv .shexport JAVA_HOME=/usr/java/jdk1.8.O_4O3.1.2. y arn-env .sh$vi yarn-env .shexport JAVA_HOME=/usr/java/jdk1.8.O_4O3.1.3. hadoop-daemon .sh修改if $HADOOP_PID_DIR = ; thenHADO

16、OP_PID_DIR=/home/hadoop/tmp fi3.2.配置文件3.2.1. core-site .xmlvpropertyhadoop.tmp.dirfile:/usr/local/hadoop/tmpv/valuefs.defaultFShdfs:/clusterfs.trash.i ntervalv/name4320fs.trash.checkpo in t. in terval60dfs.ha.fe ncin g.methodsv/namesshfe ncedfs.ha.fe nci ng.ssh.private-key-files/home/hadoop/.ssh/id_

17、rsadfs.ha.fe nci ng.ssh.c onn ect-timeout 30000v/propertyvpropertyha.zookeeper.quor um node1:2181, node2:2181, node3:2181hadoop.security.authorizati onfalsev/con figurati on322 hdfs-site.xmldfs .n ameservicesv/nameclusterv/valuedfs.ha .n ame no desclusterv/namen n1,nn 2dfs. name no de.rpc-addressclu

18、ster nn 1h1:8020v/valuedfs .n ame no de.rpc-addresscluster. nn2 h2:8020 v/propertyvpropertydfs. name no de.http-addresscluster. nn 1h1:50070dfs. name no de.http-addresscluster. nn 2h2:50070dfs. name no de.servicerpc-addresacluster. nn 1h1:53310dfs .n ame no de.servicerpc-addresacluster. nn2h2:53310d

19、fs .n ame no de.shared.edits.dirv/nameqjournal:/h1:8485;h2:8485;h3:8485/ cluster dfs.ha.automatic-failover.e nabledclusterv/nametruedfs.clie vidercluster文案大全实用文档org.apache.hadoop.hdfs.server. name no de.ha.C on figuredFailoverProxyProvid erdfs.webhdfs.e nabledv/nametruedfs.permi

20、ssi on s.e nabledv/namefalsedfs.permissi onsfalsedfs.replicati on2dfs .n ame node.n ame.dirv/namefile:/usr/local/hadoop/dfs/namedfs.data no de.data.dirv/namefile:/usr/local/hadoop/dfs/data文案大全实用文档dfs.jo urnalno de.edits.dirv/name/usr/local/hadoop/dfs/jo urnalno de v/con figurati on323 mapred-site .x

21、mlmapreduce.framework .n ameyar nmapreduce.jobhistory.addressh4:10020mapreduce.jobhistory.webapp.addressh4:19888mapred.job.reuse.jvm. nu m.tasks -1mapreduce.reduce.shuffle.parallelcopies 10v/con figurati on文案大全实用文档324 yarn-site .xmlvpropertyyar n.no dema nager.aux-servicesv/namemapreduce_shuffleyar

22、n.lo g-aggregatio n-en abletrueyar n.no dema nager.remote-app-log-dirhdfs:/cluster/var/log/hadoop-yarn/appsyar n.resourcema nager.c onn ect.retry-i nterval.msv/name2000yar n.resourcema nager.ha.e nabledv/nametrueyar n.resourcema nager.ha.automatic-failover.e nabledv/nametrueyar n.resourcema nager.ha

23、.automatic-failover.embeddedv/nametrueyar n.resourcema nager.cluster-idclusteryar n.resourcema nager.ha.rm-idsrm1,rm2yar n.resourcema nager.host name.rm1h1yar n.resourcema nager.host name.rm2h2yarn.resourcema nager.scheduler.class org.apache.hadoop.yarn.server.resourcema nager.scheduler.fair.FairSch

24、eduler v/valueyar n.resourcema nager.recovery.e nabledv/nametrueyar n.resourcema nager.zk-addressv/name node1:2181, node2:2181, node3:2181yar n.resourcema nager.store.classv/nameorg.apache.hadoop.yar n. server.resourcema nager.recovery.ZKRMStateStorev/valueyarn.app.mapreduce.am.scheduler.co nn ectio

25、 n. wait.i nterval-ms5000v/con figurati on325配置datanodevi slaves3.3.创建目录mkdir /usr/local/hadoop/dfs/datamkdir /usr/local/hadoop/dfs/namemkdir /usr/local/hadoop/dfs/jour nalnodemkdir /usr/local/hadoop/tmpmkdir /home/hadoop/tmp4.启动zk集群在 zk 节点(nodel, node2, node3)执行：zkServer.sh start三个节点都启动后查看状态，一个lead

26、er两个follower此时执行jps查看进程，启动了 QuorumPeerMain5. 启动 hadoop5.1. 启动所有节点 journalnode（h1,h2,h3 上执行）./hadoop-daem on.sh start journalnodehmdo叩間 1 sbin$ Jhadoop-daemon.sh start joumalnodestarting journalnoder logging to /usr/local/hadoop/lags/hadoop-hadoop-jou malnade-hl.out Ihadoop3hl sbin）* jps6421 JournalN

27、ode6493 Jps5.2. 格式化 hl namenodeh1上执行文案大全hdfs n ame node -format成功的话，会看到 “ successfully formatted” 和 “ Exitti ng with status 0” 的提14:&4:31 14:04:31 14:54:31 14:04:31 li；D4：31 14；04：31 14:54:31 14:04:31 1404:31 14:34:31 14:04:31 14：M;31 14:94:31 14:S4t31 14iS4：ai14 &4： 31 1:04:31 14:04:31 14：GJ4：31 14

28、:04:31 14：0d：?l 14t5A：31 1-4; 04; 3114： 04; 31 14t EX: 31 14:94:31 14:04:31 14：flA：ai 14lQ4：31LhFDIHFQINFOIhFOINFOINFOINFOutiUGSet； type - 64-lbit utiLGSet； 1.0 max memory 8 MS 8,9 MB utjl. GSat:ity- 2ZB - 1G48576 ontriMnl刊MaFSOirRturY： ACL$ enabled?niiDd?.FS0irft3ryi XAttn *nablpd? tnjsFSDire leINF

29、O metrics TopMetric si IT甲 Cflnf:計玄一ainenwdp . top Endows .ninutb 1,5.55 WO INFO INFO INFC IMFO INFO INFO INFO INFO INFD INFQ IltfO IHF0nMifiadxF3Nn#5y5tfli： Retry cclie on nmencd 15 *nbled nnenQdeTSNn&ytefn: Retry cache will use Q.Q3 of total heap 沪d retry each* utilxset； utilnGSeti vh type util GS

30、et: util,G5vt: nampnude,F$Inflge； AUotated nw SlockPcalid； BP-671X117 - 1SI，IGfl.J?!. coiuicn.Stoafl亡；Storage directory /usr/Ucal/hadoop/d+s/iane na$ been success! naniencdeSIiTageFQmaTPrctobuf： Savinq image file j us rl6caT/hal3op/= G util. .ElitBt 11: Exiting with 5上mtum & nheiDdexfJviluut nruignF

31、 ex：Deputing capacity lor map kam ewe deRetrycat he -M bit0.D79妙9=29447746生rrwory 8 MB - 273.1 KBcapacity 7T5 * 3?76B entriesi i 协：f阖帚昼科 tizm5.3.在hl上格式化ZK仅第一次需要做，在ZooKeeper集群上建立HA的相应节点，用于保存集群中NameNode的状态数据在 h1 执行：hdfs zkfc -formatZKioca tynagoop/snre/naucop/mapreQijce/ im/guice-berviexJrUnjar； /usr/

32、 iccai/naaoup/ ?nre/nacQ(jp/niapr docp -napredticeClisnt jobclint2 ,Trs，tost;. jdrur/locdl/hadoopAhara/hddoop/inapraduca/hadDQp preduce-clLent-CQre-2.7.3 jari/usr/local/hadocp/share/hadoop/iiiapreduce/hadoop -niaj reduce-client-jot f1e-2n7H3r jirusr/Iocal/hadoop/stiare/hadoop/irapreduce/hadQQj-mapne

33、duce-client-cjminan-a-yjar；/ hddoop/sha re/hidop/mapreduce/hadoup -mapreduce-client-hs-z,7.3. jarj/irsrAocal/haioop/cofiirib/cipaL7/G2/39 1495： 4& IKFO zockeeper. ZoaKseper： 17/Q3/2S 14：&5:40 IhFO zoc keeper. Zoo Keeper： 17/03/28 14:95:49 IhFO zockeeper. ZodKeeper： 17/3/2B 14；65:4。 IhFO zaokeeper.Zo

34、oKeeper: 17/&3/20 1吗：白5:40 ThFO zockseper. 7oDK?ep&r： 17/&3/2B 14：&5：4& IhFO zockeeper.ZooKeeper： 17/&3/2B 14：&5：40 IhFO iackeeper.ZooKeeper： 17/&3/2B 14；&5:4& IhFO tucketper. ZouKeejjer : 17/Q3/29 14：05：4Q IhFO 2Ck&eper. ZftoKesp&r ： 17/03/28 14：05：4& IKFO jockwper.ZooKeep&r: IectorsWatcherWitiClie

35、ntRef369D263B匚li&nt enurronment: jmwd -lib rary .path-Xu? r/local/hsdoop Client environment：lavaxio.tmpdir=/tnp Client environment ：java. compiler=KNA Client environment:os .nSTe=LinuxClient environmsnt:os arch=aird64Client environment：ds.versicn=2.6.22-642.el6.x&6_64 Client envkronment:=ro

36、otClren t envrionmtit:ubtr . f uine=/rootClient wivironment：Lj&r.dir=/usr/local/hadoDF/bin Initiating client connection, connectStrinq=nodel:21817/03/20 14,05=40 INTO zocke#per.ClieitCrixn； Opening sockftt connection to server nodel/152.160.12： 177H3/2fl 14:66:49 IWFO zcMkeeper.ClientChxn： Socket co

37、nnection established to rioiel/192,168.121,6： 17703/28 14i05i4O IhFO zockeeper.ClientCnxn： Session establishment complete on sarver nocekigi. 11 17/03/28 14；&5:40 IKFO ha.ActiveStandsyElectort Session connected.17/03/20 14,05:40 IWO harActivsStdndyElector： Successfully created /hadoop-ha/cluiter in

38、ZK.17/&328 14:05:40 ThFO rarkfper.ZoaKpppr： Session： 0K15bl2*429O800F7 closed17/Q3/28 14:05:40 IhFO zockeeper.ClieitCnxn： EventThread shut down5.4.启动 hl 的 namenode, zkfc执行：hadoop-daem on.sh start n ame no de, hadoop-daem on.sh start zkfc 此时查看进程，zkfc, name node都启动了hadoopi?nl sbin$ hadoop-daemcn.&h st

39、art nannodestarting ramGnodo, logging to /usr/local/hadoop/logs/hadoop-hadQop-nani&node-lil. out hadooph1 sbin)$ jps7765 Jps6421 JaumalNode7640 NamNodehadoopl 5bin$daemon. sh start ;kfcstarting 2kf c f Logging to /usr/local/hadoop/logs/hadoop - hadaop- zkf c -hl .out lh日doophl sbin| jps6421 JoumalNc

40、d7M9 J戸7&40 Nam eNode7819 DFSZKFailoverCont roller5.5.启动 h2 上 namenodehadaoph2 sbinls /hadoop-daedion生h start namenodeetarting nam&nadG. logging to /uGr/lacal/hadoop/logG/hadoop-hadaop-namenDde-l/h4dOD/hac/-i3 joD-naprcdu: e/hadccp/ produc/hadoop naproduc elicit jooclient 2.7.3.jar；/usr/lccal/haGoop

41、/share/hadoop/maaridjce/hadasp nipfeduce ce/hadocp -nep reduce- client-conn)on-2.7.3k j a r: /us r/locfil/hadoop/sha re/hadaop/nep reduce/had&op -map reduce- eanoles2.7.3. j ar, ths -2.73. jdr;/cont rib/cdpjcitjf-scheduler/* b jar17/C12/2fi 14；45；12 2NFO nsmenode.MameDde； sgistersd UMX signal hardle

42、rs -far TERflj l-UPr UT 17/03Z2G 14；45； 13 :MO nan enode. NaneMode； c rsateNarelScde -bcotst rap Standby About to bdoiqirap standbybtinsvrvic 10：D BO7 fLjn： clustsOttier Nunonode ID： Cther NNs HTTP addres?； Other fW5 1PC address : HcKieftpdce ID： Block pool ID: Cluster ID: Layout version: isUpqradeFindlized:nnlhttp；/hl;50078hl/57:533131111443281時-briMLir” i y2.iti.i2i.i67- gyt曲自比聲jUD f f d创)7 7 曲他-4妁6-詰8a 751殆呛餌晒7 63tnie17/G3/2B 14:45：it TrjFO cmfln.stors：jj： Sts -age directory /usr/local/hidonp/dfs/naw? has been succes