大数据技术实验一平台搭建

《大数据技术原理与应用》实验报告一题目：安装Hadoop姓名：高聪江班级：大数据1533学号：2015005677日期：2017.11.01实验环境：笔记本电脑一厶Archlinux虚拟机3台Hadoop2-7-10Java1-8-0实验内容与完成情况：下载相关软件已完成安装虚拟机已完成配置环境变量已完成安装Hadoop已完成出现的问题：01虚拟机运行软件选择02虚拟机选择03虚拟机的分盘出现问题04虚拟机命令不被识别05Hadoop版本选择06Hadoop无法正常启动07结束安装后活节点数是0解决方案（列出遇到的问题和解决办法，列出没有解决的问题）：01虚拟机运行软件选择使用了VirtualBox02虚拟机选择使用了Archlinux03虚拟机的分盘出现问题再分了一次，成功解决04虚拟机命令不被识别系统的环境变量出现问题，修改配置文件解决05Hadoop版本选择选择了2-7-10这一比较广泛的版本Hadoop06Hadoop无法正常启动由于格式化节点的时候没有先把生成文件都删除07结束安装后活节点数是0这个问题原因一直没有找到，重装解决的报告正文1.实验内容与步骤本实验主要内容和步骤如下:1・1•实验内容

Hadoop集群搭建（由于伪分布比较简单，所以实验报告描述集群搭建），WorldCount实例测试。1.2.实验步骤1＞下载相关软件。主要是Java，VirtualBox，Hadoop安装包，虚拟机的镜像文件。2＞安装ArchLinux虚拟机（由于此发行版本比较洁净，系统体积小，运行比较流畅）安装结果如图：AirkLiwK4-12」3-l-fiKII«thjtkAirkLiwK4-12」3-l-fiKII«thjtkrw*仙曲ssmitM：Ls3t冋in；tied问10t：(£"5Dont坷】WtnniE1zbf-甌NEnxitfcwif血庙1"Ji密IxtL球少l対]id(rooWcflniEtenfll"11AkpclD皿:hdlieu：losdniudevkhimi汩lnt^rT«C吕Ihmtatdrrler忆Mtaluckyrcuitd・chill胡(TWl4<4Mi|^:lM；n^1"]■f^|iG<jrg?hc^l.5ListIcqin：Ihu旳uZQ2：的”15Z»I7E解Uli何的睛血町1[网Wcaif处庙1^11taillogoulJDtJElf'lfallComBctiantonngsherAhclosed.[fixil#ctHif^heiiBl^11tstmrywhe保1IM：tellerLas-tlo^ln：IhiiNtw26fi：le：Lb3&I?hti'jEpimliinEmfiUkLifihrnAZ~llajlIcquul(ZwccUantocon闾whc似clwefl.CrQfltfCjinfi.l<!n^1~ll.s.z..iHi.：.jjfcAd也v荀<“灯亠“L.s.Ltl.H.J1AT■LQ.JdlL.J3MAW.：j|3＞进行Hadoop集群安装＞＞启用网络并安装SSH和JDK＞＞配置SSH免密登录rchLinux4,1Z-13-1-AKCH(ttyl)conjslienGllogin:rootPassword:怙航login:WedMou101:08：50onttyl[roDtScongslieiiOl"JKsystemct1startsshd[rootecangshenOl~litdhcpcddeu:loadedu.deuuuinteffaceshaueacarrierforkedtobackgroundscliiIdpidZ83[rootecongshenOl~litsshcon^shenOLLastlogin：ThuNouZ02:09:152017[rootPcangshenOl~IBexit.logoutConnectiontacongsliEnOlclosed.rrootPcongsheiiOl"litsshcongshEn02Lastlogin：ThuNouZ02;IO;16Z017[rootteongshenOZlitexitlogoutConnectiontocungsbpn02closedTFroot@congshe)iGllit>>配置JAVA环境变量.rootPcongsheiiUlJlljava-versionqpenjdkuersionUL8.0^144"IpcnJDKBuntineEnuironnent(buildl.8»0_l14-bOl)IpenJBK64-BitSeruerUN(build25.144b孔nixednode)TootecongshenOl“]It$JftUA_HOHE/bin/jaua-versionqpenjdkversion师1・8・0」44"fpcnJDKRuntimcEnuirQnncut(build1.乩01斛401)IpenJDK64-BitSeruerUH(build25.1H-b91>nixednode)显示上图说明JAVA环境变量已经配置完成。>>安装Hadoop[roatKongsheiwlIthadoopuersionHAdoopZ.7,1Sukersionlittps7/shuOgit-wip-us.apache-org/repos/dsMiadoop.git-rcd?15ele8l5dO1314fi2aflb7391586cl7572Ba282CompiledbykshuachkonZO17-O8-91J09：23ZCmpiledwithprotoc2,5』Fronsourcewithchecksun50b046B318bke9M24dc467b7cell4BThiscoiwnduasmusing/usr/1oca1/hadoof/share/Woop/copnoi/ludoop-connDn^安装结束如上图>>配置Hadoop环境变量在~/.bashrc文件中添加如下内容:exportJHUAHUlfcz/usr/11b/jun/jaua-K-openexportHADOQTHOffi^usr/loul/hdDO}exportHOT_lKSTftLL=SHADOOP_HOHEexportHftDOOPJflFRED.HOHE^.HftDUUFHil卄eitportHADOOF_MAPRED_HOHE=JllflliOO]'HOriEexportHADOOP_CDfflOH_HOHE=5HftDOOP_HOHEexportHftDOOF_HDrS_fflE=；i^OOP_HdHEexportYARNHfflE=$HADDDPHONEexportHAD00F_C0tmi_L[BHATtUE»Il=^DOOrHH/lib/nfttiueexport:SinDDOrjIME/sbii»：$HflDOOP_HOHE/binexportHftDOOFJPTS^-Djaua.1il>rdry.patli^HftDOOP.HOnE/lib：jHftDOOF_CDnnOI1_LIB_NAIIUEJIF*1保存并执行$source〜/.bashre使其生效。>>修改Hadoop配置文件进入cd/usr/local/hadoop/etc/hadoop/修改core-site.xml为:corfijuration}<property><nane>hatlo叩・tnp.dir</nftne>(uttluc>fiIc：/usr/local/liadoop/tnp</uakc)(description>ftbaseforothertenporarydirectories.(/descriptian{^property}(property}<niUK>fs.dEfaultFS<AwE>1ue>hdfs://cangshEitO1-S(M0</ia1ue></property>^property)<HHe>io.File.buffer.sizE</nane)<ua]ue>131O7Z</ualuE)(/property>修改hdfs-site.xml为:Ccaofijuration><prapert^><nitFie>dfs,replic4tion</nane>(/property><property><naM>dfs.nineHole.none.dir</Mnt><ualne>file；mr/lxaI/hadoop/tnp/dfs/nane</ualue></property><pf[)perty><nane>lfsJitanadeJala.dir(/nine)<ualue>file:/ur/local/hiidoop/tnp/drs/dflta<Amlue></property><prapM*ty><naie>4fs.blocksize</i*ne><wlue>256i</wilie>(/property)>>启动Hadoop输入./bin/hdfsnamenode-format来格式化输入./sbin/start-dfs.sh来启动守护进程rootwcongshenuihaloopJflysbivstart-dis.shbash;./slin/start-dfsHsh;HosuchfileordirectoryTontOcongsheiidlhdcnplflstart-dfs-sh7/11/GZ0341:01UftHNutiJ.NatiucCodElwder：IInahl&toLoadnatiue-hadooplibrayforyourplatforn.,,usingbuiltin-jayaclassesutereapplicablenaneiiudeson(coiigshenOl]:ongshen01'startinjnanemdc..luggingto/usr/locaI/hfldoapi/hadDap_1ogs/root/hadiop-root-niineiiQde-CDnc[sheTiGloutungshen^：startinj血廿1和血』Loggingto/usr/lociil/hadoop/]ogs/ha4oop-root-datmde-cvngshcn03,ouitingshenQS:starting血tdin血』loggingto/tisr/local/hadoap/logs/hadoop-root-ditnode-cangslien&Z.out:hrtingsecondarynanenodES[D.0.4.9]1.0.0.0:stcandarynanewoierunningasprocess832.Stopitfirst,.7/11/0203:11:21UftRNutil,NdtiueCoMoider:Uiktb1&toloadnatiue-hfldooplibraIformgDhtfoHL"貯imrbailtin-X^ch^esulierewulisblE>>查看进程输入jpsruuLW：u«iysncnuiri(wuupjfljjj^195Jps086SecunhryNaReNode900NaneNode

251JpiwtagshflK"Iln251JpiwtagshflK"Ilnt伽”聊l/hiMpdogs/r«Dtft：Di^M2\lm-rfusMlool/俪oop/tnpf血伽询诫半jp[raotfcDNIShO]就轉jp$IJVit...>>进入网页查看状态M刊口ueqcIuLn"fori3tLDni-ChromLiJiwVDX7}\sPOG'JP—匾E&00-192.168.56.101:50070/dfshealth.hftmltttah-dvervie^v肯■INonHlEgpIM亡moryui呂Ed4D.E7MIEor斗L5(MBCommitedNonHeapMemcry.MaxNonneapMemoryis-1B,CcnliguredC^paicity:29.4GBDPSU$^d：48KB(0%)N^hDFS6.83GBDF5Rcrnairiiing"21.04GB(71.55%)BlockPoolUsed:4BKB(&%}DataNodesusages%[Min/Median/Max/stdDev}:0.00%/0.00^70.00%J0.00%LiveNodes2(Decommissioned:创DeadNicieso(Decommissioned：0}Decorn<mi&slonin9Nodes0lotalDatanodeVolumeFailuresa(ab)MumberofUnd君r・Replic^t电<1Blocks0NumberofBlocksPendingDeletion0BlockDeletionStartTime1V2/2O17P11:25^9AM>4WorldCount实例测试<1创建用户目录。<2将本地input文件夹中的数据上传到HDFS的input文件夹中。<3运行命令来执行字数统计测试样例

[hadoop@匚ong5henhadoop]smkdir・/input[hadoop@匚ongsjh亡mhadoop]Scp./etc/hadoop/*.xml・1input[hiadoop@匚□ngshenhadoop]sh□doopdfs-put./inputDCPRECATEDiUseofthisscripttoexecutehdfscommandisdepredated.In5teadusethehdfs匚ommandforit.17/11/0515;L144WARNutiINativeCodeLoad?rUnabletoloadnative-had□opIibraryFqryourpIatformusingbUiItin-jawa匚1e5whErbaPP1iEab1e:ha詁o□•匚□ngshenhad□口P]Shado口PjarJU5F/1□匚a*/had□dp/eharefhadu□p/mapredLTcefhad□口p-rnapr&du匚a-amp