实验2-熟悉常用的HDFS操作

实验2熟悉常用的HDFS操作1实验目的理解HDFS在Hadoop体系结构中的角色；熟练使用HDFS操作常用的Shell命令；熟悉HDFS操作常用的JavaAPI。2实验平台操作系统：LinuxHadoop版本：2.6.0或以上版本JDK版本：或以上版本JavaIDE：Eclipse3实验内容和要求1.编程实现以下指定功能，并利用Hadoop提供的Shell命令完成相同任务：（1）向HDFS中上传任意文本文件，如果指定的文件在HDFS中已经存在，由用户指定是追加到原有文件末尾还是覆盖原有的文件；wen^ubuntu:/usr/locdl/hadoop/bin$./hdfsdfs-test-etext.txtwengubuntu:/usr/locaVha4oap/bin$echo$?127w亡n@ubuntu;/usr/locdl/hddDop/bi.n$*/hdfsdfs-appendloFilelocal*txt叩pendToFile:/usr/local/hadoop/bin/local.txt（疫有那个文件或目录）NenSubuiitu:/usr/local/hadoop/bin$./hdfsdfs-cp-ffile:///hoHe/hadoop/local11xttext*txt匚p:'file:/j/hone/hadoop/lo匚廿Nosuchfileordirectory.'LProblems矽」avado匚陽DeclarationSConsole阅<tefminated>HDFSApi[JavaApplication]/ujf/ljb/jvnZiava-7-openjdk-amd64/birVjava（Dec15,20164:54;2?iAM）log4j：>j^rnNaappenderEcouldbefoundforlogger{org.apache.hadoop.conf.configuration.deprecat:log4j:WARNPleaseinitializethelofl4jsystanproperly.Iog4j:warnseehttp://Logging.apache.«rg/log4j/1.2/faq.htmltnoconfigformoreinfo./user/hadoop/text.txt已存在，/hame/hadaop/text.tt已追如至/user/hadocp/text.txt（2）从HDFS中下载指定文件，如果本地文件与要下载的文件名称相同，则自动对下载的文件重命名；hadaDp^ubuntuT/usr/local^hadoop/bln^./hdfsdfs-test-eftl_e:///home/hadoop/text*txthadoopiLibuntu:/usr/l.ocal./hadcopybtn$./hdfsdfs-copyloLocaltext.txtr/text2.txthadoop^ubuntui/usr/lDcal/hadGopybln^*/hdfsdfs-copyloLocaltext*txt,/test>txt』IH讶尸耳—ff■JkhFK 書■■亡 1 F*Problems曲Javadoc凰Declaration旦Console辽、^<；terminated>HDFSApi[JavaApplication]/u5r/lib/jvm/ja/a-7-openjdk-amd&4/bin/java(Dec15,20lc-g4j:WARNPleaseinitializetheLog4jsystenproperly.Iog4j:WARNSeehttp：//laggingapache.arg/log^j/I..2/faq.html^noconfirjtorworeinti/home/hadoop/te>t.txt已將垂新命名为：/hame/hadoop/text.txt_0将HDFS中指定文件的内容输出到终端中；hadoop^kibuntLi^u^r/local^ti^dDop/bln^./hdfsdfs-cattext*txthelloliadoop%Problems@Javad□[:岭Declardtian旦ConsoleS3vtfqninatfd>^DFSApi[」弓杓岛pplit吐ion]/usr/Lib/jvm/java-7-openjdk-amd64/bir^java(Dec15,20154:58：'Lag4j：WARNPleaseinitializetheLog4j&ystemproperly.读取文件：/user/hadoop/text.txtlag4j:WARNSeehttp：///log4j/I.2/faq.html#naconfigfarmoreinfa.hellohadoop显示HDFS中指定的文件的读写权限、大小、创建时间、路径等信息；hadoop@ubuntu:/usr/local/hadcop/bln$./hdfsdfs text.txt-1hddoopsupergroup 13201C-12'1504:54text*txt1口g4j:WARNNoaopenderscouldbefoundforLoggerIerg.apache.hadoop.conf.Configuration.(1口g4j:WARNPleasEinitializethelag4jsystEmoroperLy.la^4j:WARNSeehttp：///log4j/l.2/faq.html#noconfigformoreinfo.踊至:hdls：//localhost：9000/user/hadoop/text.txt权限：rw-r--r--大<h：13时1可：2016-12-1504:54:37（5）给定HDFS中某一个目录，输出该目录下的所有文件的读写权限、大小、创建时间、路径等信息，如果该文件是目录，则递归输出该目录下所有文件相关信息；rhadcapguhuntu:/u£r/local/hadcop/btn$ hdfsdfs-Is-R-h/user/hadoop!-rw-T'-r-- 1hadoopsupergroup 132016-12-15。斗；54/user/hadoop/text*tIxt<terminatgd=HDFSApi[JavaApplication]应r/Liap^ojdk~amd64/bin/j□问(。改巧log^jrWARNNdappenrierscouldbefoundfnrlogger(nrg.apaer已-11日(10口卩-(:口11十一亡[311千:1§1]「日11_口口-4已卩口lag^lj:WARNPleasein力lizethelog^jsystempnoperly.Ioy4j:WARNSeehttp;///log4j71■2/faq«htnl#noconfigformoreinfo.(递村[读取目录下所育文件的信忌;：/user/hadaopS3经:hdfs://localhos-z：9006/user/hadoop/text.txt权FS：rw-r--r--大小：13时间：2016121504:54:37提供一个HDFS内的文件的路径，对该文件进行创建和删除操作。如果文件所在目录不存在，则自动创建目录；jhadoop^ubuntu:/usr/local^hadcop/bln$-/hdfsdfs-rmdirl/dirZ/filename|rn:'dirl/dir^/ftlename':Nosuchfileordirectory昙Problemi住」avadoc鳳□eclararion曰匸onsoleS3<t&rminatedHDFSApi[JavaApplica11&n.]/usi/lib/jvm/jjava-7-&perjdk-amd6^1/bin/java(Dec1320165:06:2!lag^j:WAftNNdappendsrscou'.dbefoundforlogger(org.apache.hadoop.conf.ConfigLration.c1口g4j:wafinPleaseLritiali?ethelog4jsysteraproperl/.lag^j:WARNSeehttp：//lagging.apache.arg/lag^j/1.2/faq.html#roconfigtornoneinto.创亘文件决：/user/hadoop/mput创理貉径:/user/hadoop/ln叫.txt提供一个HDFS的目录的路径，对该目录进行创建和删除操作。创建目录时，如果目录文件所在目录不存在则自动创建相应目录；删除目录时，由用户指定当该目录不为空时是否还删除该目录;IhadccpQubuntii:/usr/local/hadoop/bln$«/hdfsdfs-nkdtr-pdtn/dlrzjhadoop^ijbuntu:/usrflocal/hadoop/bin$*/hdfsdfs-rndirdirl/dir2hadoopOubuartu:/usr/local/hadoop)binS+/hdfsdfs-rn-Rdirl/dirZj「(i; ：Mu各uthfiL亡q「di.「巳et0「y暑Problems|®Javadoc為Declaration旦Console韶<terminaLed>HDFSApi[JavaApplication]AJ5「/litj/j¥nVjava-7-openjclk-anicj&4fbiMa询(D亡t:15,20165Log4j:WARNNoapperderscouldbefoundforlogger(ory.apaclie.haJoop.conf.匸aiiflguiatlog4j:WARNPleaseinitializetheLog4jsystemproperly.Log4j:WARNSeehttp：///log4j/1.2/faq.html#noconfigformoreinfa.目奈禾为空r不fll踪：/user/hadoop/input向HDFS中指定的文件追加内容，由用户指定内容追加到原有文件的开头或结尾；hadcop^ubuntuj/us.r/local/hddQop/bin$*/hdfsdfs 叩endToFtlelocal.txttext*txtappendToFilet/usr/local/hadoop/bin/local.txt(Nosuchfileordirectory)hadoDp@ubuntu:/usr/lDcal/hadoop/bin$*/hdfsdfs-gettextktxtgeti'text*tst':Filee^ist吕hadoDp@ubjntu:/u£r/lDC3l/h3doop/bLn$，/cattext,txtnalocal.txtbosh:*/cat:NosuchfileardtrectoryhadoopQubuntu:/«sr/local/hadoop/bins+/hdfsdfs-copv^ronLocal-ftext.txttext,txt豐Problems&」avadoc風DeclarationJSConsole淤■tterminated>HDFSApi[JavaApplication]/us「/lib/jvm/ja*a-了-0p亡njHk-nmdi砧旳a(Dec15,Log4j:WARNNoappenderscouldbefoun口forlogger(org.apache.hadoop.conf.ConfiguraLogi4j:WARNPleaseinitializethelog4jsystemproperly.Log4j:WARNSeehttp：///log4j/I.2/faq.html#noconfigtarmoreinfo.巳追IB內容到盘件未庭fus已17had。叩灯亡巩.txt(9)删除HDFS中指定的文件;|hadoop@ubuntu:/usr/local/hadoop/bin$../hdfsdfs-rntext*txt16/12/1505:14:51INFOfs^TrashPolicyDefault:Nanenodetrashconfiguratian:DelettonInterval=0nitnutes,EnptlerIriterval=0minutesDeletedtext.txt由于已经删除，所以会显示操作失败。<cerminacea>HDr^Ap1LJavaApputia口orj/usryiiDfjYinfjaYa-f-opEnjQK-anioM/DifVjd^aiu已1g约lag4j:WARNNoappender'5couldbefoundlorlogger(arg.apache.hadoop.conf.Canfiguralog4j:WARNPleaseinitiEslizethelog斗jsystemproperly.Iag4j:WARNSeehttp://logging.apache.Drg/log4j/L.2/faq.htnl^nocontigformareinfo.操柞失败(文件不存在或制降失败)(10)删除HDFS中指定的目录，由用户指定目录中如果存在文件时是否删除目录;

由于前面已经删除，所以显示无该文件.Ihsdcop^ubuntu:/usr/local/hadoop/bin^>/hdfsdfs-rmdirdirl/dir2jrndtr:ditrl/dtr2':NosuchfileordirectoryEhedcop^ubuntu:/usr/local/ha(loop/bin$>/hdfsdfs-rn-Rdirl/dir2|rn:dtrl/dir21:Nosuchrileordirectory-8Problems®Javadoc嗚Declaration□Console滋1、<terminsted^HDFSApi[JavaApplicationl/usr/lib/jvm/java-7-openjJk-amd64/birVja/a(Dec15r2O1<55;19;2log4j:WARNNod叩enderscouldhefoundforloggei(org,apache.hadoop.conf.configurationrlog+j:warnPleaseinitializeThelogAjsystemproperly.Log4j:WARMSeehttp：//logging.apache.Drg/log4j/l.2/laq.htnl^naconfigformoreinfo.目录不为空「不删煤(11)在HDFS中，将文件从源路径移动到目的路径。hadoop^ubuntu:/usr/local/hadoop/b'InS./hdfsdfs-nvtext.txttext2.txt移动成功：utErminatEcinHDFSApiL」叭九pplicati0n]fuM[7LibfjvimZjaw-'7-cipefijdk-a[nci64/bin/jaum(Dec15,.20165:26:11AM) 1og4j:warnNoappendsrscnji.dbefnjndforlogger(org.a阳hadoop,c叩匸「onfiguration.cl即reca-i.加].Iog4j；WARNPleaseinitializetheLog4jaysteaproperl/.Log4j:WARNSeehttpi/Zloggi'ig.apache.org/log^j/1.2/faq.hta^noconfigformoreinfo.将立件ihdfs:///useryHiadoop/te«t,txt移朋亦hdfs:///user/hadoop/new”txt2.编程实现一个类“MyFSDataInputStream”，该类继承“”，要求如下：实现按行读取HDFS中指定文件的方法“readLine()”，如果读到文件末尾，则返回空，否则返回文件一行的文本。进行文档读取如下:屍Problems@Javadoc冏Declaration旦Console爲<terminated>MyF5Datainputstream[JavaAppli匚日Lion]/usr/Ub/jvm/java-7-openjdk-amd64/bin/javaLog4j:WARNNdappenderscouldbefou