实验2-熟悉常用的HDFS操作

1、实验2.熟悉常用的HDF蹶作实验2熟悉常用的HDFS操作1 实验目的1. 理解HDFS在Hadoop体系结构中的角色;2. 熟练使用HDFS操作常用的Shell命令；3. 熟悉HDFS操作常用的JavaAPI。2 实验平台操作系统：LinuxHadoop版本：2.6.0或以上版本JDK版本：1.6或以上版本1. JavaIDE:Eclipse实验内容和要求编程实现以下指定功能，并利用Hadoop提供的Shell命令完成相同任务：向HDFS中上传任意文本文件，如果指定的文件在HDFS中已经存在，由用户指定是追加到原有文件末尾还是覆盖原有的文件；wenubuntu:/usr/local/hadoo

2、p/btn$./hdfsdfs-test-etext.txtwenaubuntui/usr/local/hadDop/btrSJhdfsdfsppendloFllelocal.txttext,txtappendToFile;/usr/local/hadDQp/bin/local.txt做有那个文件或目录）Menubuntu:/usr/local/hadDop/btn$./hdfsdfs-cp*ffile:/home/h3doop/localttxttext,txtcp:1file:/hone/hadoop/local.txt1:NosuchfileordirectoryPrsbLemsJava

3、docDeclarationCon&ole'Sl'erminfll'riHDF5ApiJavaAplkatrnnjAjs(7lit)/j'yrravti-7-opFriidknfiiTidD4/biryjavfl(Cec15,?C164S4：?8AM)loq4；na圈hqappnderjcpLLdbefoundfurlogger(orgpapachedoop.conf-ConTiquration.depr«catlogj:WARNPleaseinitialisethelog4jsystenproperly,lc»q4；:warjjse

4、ehttp：/loqqinq1apa<lie1orq/l,og4j/rj.2/fraqhhtiTr#nQCflnflgforinfp./user/hadoop/texl.txt已存在，/hoiie/hacInnp/teKt.tMt至/user/hadDoa/'texttxt从HDFS中下载指定文件，如果本地文件与要下载的文件名称相同，则自动对下载的文件重命名；hadoop世11勤11口史11£/11$./'10匚31./11眉"0。2/勤111$./hdf£dfs-tet-efile：/fwfteyhadoop/text.txthadoopQ

5、ubuntut/usr/loca'L/hadoop/bln$./hdf£df£-copyToLocaLtext.txt,/text2,txthadoopubuntu:/usr/Localhadocp；blnS，/hdfsdfs-copyToLocaltext*txt./text.txtk.rJncjau.Lrii/f/I1fJk.；史1fTu.Problems疝avadxE,DedarationCcnole网<temii-iflted>HDFSApiA|ppLicdtion/jsr/lib/jvm/jdd-?-operjdk-jmd54/biryjdVf

6、l(Dec15,20lcg4j:WARNPleaseinitializethelog4jsystenproperly.Iog4j:WARNSeehttp：/logging,/LQg4j/!.2/taq.htril#noconfigtorloreinf/kome/hadoop/te?t.txt已存在.(1) 将圭新名名为：/MrreFhadDDp/teid:.txtG下韩完成将HDFS中指定文件的内容输出到终端中;hadoopgubuntu:/uisr/local/iadoop/bln$./hdfsdfs-cattext*txthellohadoop土Protjlems莅Ja

7、vadoc底Dedaraticn9ConsoleES<terminated>HDFSApiJavaApplication/usr/Lib/j¥rn/java-7-apenjdk-anid&4/bin/java(Dec侣驷Laq4j：WARNPleaseinitializetliclog4js/steaproperly.应取丈忡：/jser/hadoop/text.txtLog4j:WARNSeehttp:/L/Log4j/I.2/1aq.html#nDConfigformoreinfo.hfillihadetip建取完成显示HDF

8、S中指定的文件的读写权限、大小、创建时间、路径等信息;hdoap$uburtu：/u$rylocal/n$./hdfsdfs-Is-htext.t)ct-rw-r-r-1hddoopsupergroup132016-12-1504;54text.txtlag4j:WARNNoaoaenderscouLdbetcundforLogger(Drg.apache.hadcap.ccnf.Configuration,ilqg4j:WARNPleaseinitializethelcg4jsystempros&rly;WARNSechttp；/logging.apsche.argAogj/1.2/

9、faq.html/poconfigformoreinfo.(2) Wt:hdfs：/lacalhost：9O9&fu5er/hadaop/te;(i,txt枚珞：大小：is时间：2»1G-13-1564:54:37给定HDFS中某一个目录，输出该目录下的所有文件的读写权限、大小、创建时间、路径等信息，如果该文件是目录,则递归输出该目录下所有文件相关信息;Rhadooputkuntu:/usr/locL/hadcopybtnJhdfsdfs-Ls-A-h/useryhadoop!*rw*r*-r*-1hadoopsupergroup132016-12*lb/s&r/ha

10、doop/texttIxtterminatedHDFSApiJdvAAppliticn/usr/lib/jvm/java-7-apenjdk-amd64/biryjava(Deci5r20155;D2;40AMloglj:WARNNoappenderscouldbeFoundfcrlogger(Qrg.apachE.hadQap.cDnf.ConfigUrati.Dn.depr:WARNPleaseiniticilizeth?Iog4jprojjerly.logj；WARNSethttp:/Logging,apacherorgi/lDg4j/L2/faq.htrrl+noconfigforno

11、reinfo.(a*3>iW曰染二用有文炸的样息：/user/hadoopS®f?:hdt5：/localhost:900G/user/hadoop/text.txt枳匣:rw-r-r-E：11(3) 间间：3B»12IS64:54:37提供一个HDFS内的文件的路径，对该文件进行创建和删除操作。如果文件所在目录不存在，则自动创建目录;hadoopubvntu:/u£r/local./h>doop/bin$>/hdfsdfs-rmdtirl/dtr2/filenamem:'dtrl/dirZ/rilerdne1:Nosuchfileord

12、irectoryProblemsJjdac®：DetkinHonQConsolefS<terminat.ed>HDFSApiJava山叫)氐3口0可，鹏1/1旧/叫1/"咨-70>如业岫mdg/bi凹池a(Decisr2016s?oe：z<Icgj:WARNMoappenderscouldbefoundfcrlegger(org.apache.hodoop.conf<ConfigorotionTJlog4j；WARMPleaseinitializethelng4jsysteiiproperly.Log4j；WAKNSeehttp：/loggin

13、/lcq4j/l.2/faq.htnl#ricconfigformoreinfo.(4) 创隆女件奖：/ser/htjdGop/l»ipuc创购径：/user/hadoop/inpur/Text.t>r提供一个HDFS的目录的路径，对该目录进行创建和删除操作。创建目录时，如果目录文件所在目录不存在则自动创建相应目录；删除目录时，由用户指定当该目录不为空时是否还删除该目录；hddoopnbuntu:/usr/lacat/hadoap/blnS*/hdfsdTs-pikdtr-pdtri/dlrihdcopubuntu:/usr/locl/hdcp;btn$

14、*/hdfsdfs-rdtrdirl/dtr2hadoopubuntu：/usr/Local/hadoop/bLnS»/hdfSdfs-rn-Rdlrl/dtr2H：2WNomehflit?<jdLegtsv*_Prebkms伊JavadocDcclaraticriConsoleS3<termnaLed>HDFSApiJavaApplication)/usr/lit/jvm/java-7-openjdk-anid64/bin/java(Dec15,20165Loq4j:warnnoappenderscouldbefoundforlogger(org.apache.h

15、adoop.conf.corHquratlog4j：WARNPleaseinitializetheIog4s/stemprooerly,Log4j：WARNSeehttp：/1ogging.apachecrgog4j/1.2ffaq.html*noconfigformoreinfo.目歧禾为空,不携:Aiser/hadcop/input(5) 向HDFS中指定的文件追加内容，由用户指定内容追加到原有文件的开头或结尾；hadpopAubuntu：/u!>r/lacdL7hdduup/bLii$«/hdfsdfs-ppendToFllelocal.txttexttxtppendTo

16、Fllet/usr/LocaL/hadoop/bln/Local.txt(Nosuchftleordtircctory)hadoopALihuntu：/usr'/lacdl.yhdduup/bln$+/hdf5dfs-gettLxt.txtgot:text*txt1:Fileexistshadoopiibuntu;/usr/lacdl./h<idoupybln$,/cattext+txt>>local-txt“cM:nosuchfileordirectoryhddo0>jubuntui/i>sr/la<dl/hdduQp/btii$./hdfsdfs

17、-copyFropiLocdl-ftext»txttext*txt?.ProWems回Javadoc凰DecLarabarJF导Console£3<terminatedHDFSApiJavaApplicationyusr/lib/jvm/java7-open.dkamd6/bin；jaa(Dec15,2016Log4:WAR.MHoappendscouldbefauridfarlogger(org.apache.hadoop.caul.ConfiguraLog4：WKRNPleaseiniildliztthelog4jsystemprouerly.Log4:WARNS

18、eehttp：/loggirgapache.crg/lag4j/1.2/faq.htmlnoconigformoreinfo.rliiHl?14/userfhadnop/tPxt.txt删除HDFS中指定的文件;h3do«pvbuntu:/usr/lool/hodoop/bin$*/hdfsdfs-rntext»txt16/12/1505;14;51INFOfs+TrashPolicyDefault:Hanenodetrashconfiguration:OelettonintervdV=sMtnutes,tncrval=0minutes-Deletedtext*t>t

19、由于已经删除，所以会显示操作失败。grEiriacEghum。mAppMauDn/U5r7iiD，jm，j3vm-H)pew<jK-amQ4/Diwjavatlcg4j:WARNNoappendsrscouldbefcundforloggerorg.apache.hadoop.c&nf.ConfiguraLog4j:wakmPleaseinitializeTheRg4jsystemproperly.Iog4j:WARMSee/log4j/1.2/faq.htnloconfigft>rmoreinfo.操作关敝C女件不存在或制院

20、关殴)删除HDFS中指定的目录，由用户指定目录中如果存在文件时是否删除目录；由于前面已经删除，所以显示无该文件hadoopuburtu:/usr/1.cc3L/hi(icop/btn$./hdfsdfs-rmdtrdtri/dir2mdtr:kdtri/dtrz':nosuchrileordirectoryIhadaopubuntu:/u£r/locsl/hidoop/biin$*/hdfsdfs*rn-Rdirl/dir2Irm:PLUdw':nd$uchfLLe心;dlrectorv%Problems板Javadoc忌DeclarationConsoleS3III

21、<terminated>HQF5ApiJavaApplication/u5r/Lib/jviTi/javd-7-cipenjdk-flind64/bin/javd(Dec15,20l65;1;2lag4jrWftHMNoappeuderscouldbefoundforlogger(org.aiache.hadcop.corf,confIguratIon.lac|4j:warmPleaseinitially?Thelog4jsysTenproperly,l.t)g4j：WARMhttp：/Ingging,apachenrg/1ogej/1r7/taq,fitnlJtnacnnfigf

22、ormoreInfn.M录不为理，不僻-在HDFS中，将文件从源路径移动到目的路径。hadooputNintai/usr/locaI/hadoop/btn$/hdfsdfs-rvtext.txttext2.txt移动成功：terminated>HDrSApHJauiApplicaticn/usr/lb/wm/java-7'Cpenjdk-imd64/bin/.avii(Dec15,20155：2B：11AM)locjj:warnNoapppnder?rsuldbefounlfarlogger(org.apache,hriocp.conf.ccnfigurationdpprecal

23、ionLoq4:dAWJPleaseinitiatizethelag4steaprajerly.Iog4j:WARNSechttp:/lodging>apache.srg/j/1.2/fhtalfnotonffornoreInfo.2. 犒立信hdfs：/User/hdidOOp/ti!xl.txt精出#hdf$I/user/hdno|/ncwTxt编程实现一个类“MyFSDataInputStream”,该类仑也承"org.apache.hadoop.fs.FSDataInputStream"，要求如下：实现按行读取HDFS中指定文件的方法“readLine()",如果读到文件末尾，则返回空，否则返回文件一行的文本。进行文档读取如下：ProblemsJevodotDcclation2Console悝<cerminaed>mvfsDacampjestreatjavaAnpli匚就on/usr/lib；jvm；java-7-opendkamd04/bin/jevalog4j；WARNNoappenderscouldbefoundforlogger(ergnapache.hadoop.conf.Configura