实验2-熟悉常用的HDFS操作

1、精品文档7欢在下载实验2熟悉常用的HDFSB作1实验目的1 .理解HDFS Hadoop体系结构中的角色；2 .熟练使用HDFSt作常用的Shell命令；3 . 熟悉HDFSB作常用的Java API。2实验平台操作系统：LinuxHadoop版本：2.6.0或以上版本JDK版本：1.6或以上版本Java IDE : Eclipse3实验内容和要求1 .编程实现以下指定功能，并利用 Hadoop提供的Shell命令完成相同任务：(1) 向HDFS中上传任意文本文件， 如果指定的文件在 HDFS43已经存在，由用户指定 是追加到原有文件末尾还是覆盖原有的文件；wenSabunta:/usr/lo

2、Gal/hadocp/binS ./hdfs dfs -test e text»txtMnflubuntu:/usr/loca'L/hadoop/btn$ echo $? 127福罐ubuntiK/n八oel/h己加印/binS Jhdfs dfs appendloFlle local.txt text*txt叩pendTQFilE：川5171。口1独期。印皿11/1。匚日1,1式(笈有那个文件或目录)wenSubuntu:/usr/Uc«l/hadoop/bin$ »/hdfs dfs -cp -f file:/hone/hadoop/ocal.t xt

3、 text,txtup: "file;/hvf/hadQop/l。匚aLtxt：: No file odiKtoy；ProbLtms H Jvadlac O«<laration O Coniolt 锯<tefminal-'=ri> HDFApi Java Applkatirinj Aj5(/|itjvmavti-7-oppnjdk-fimd64/biryjflva (De<1 A ?C164 S4：?8 AM) loq4：narn no appenaer 匚。_Ld be found *or loqqer (qrq.apache,hadoop

4、.conf.conTiguration.depnecat log4j ;RN Please in 111 a I j th? Ing4j systen properly,loq4 j ：nakn see hTtp：/logqinq .apache .org/Loj/j .2/raqhlitil#noconfig for no re info./use r/hadoDp/tesr .txt m存在，hciiiEfhacknp，tE：Kt_tMt miGtf至 /user/hadDO3<teit txt(2) 从HDFSf下载指定文件，如果本地文件与要下载的文件名称相同，则自动对下 载的文件

5、重命名；-te5t -e file：hadoop/tex-copyIoLocal text.txt ,7text2,tx-copyToLocal text*txt ./text.txtbiadoDpubunTut/usr/local/hadGop/btn$ ./hdfs dfs t.txthatloopflubuntu 1/usr/localhadoop； bLn , /hdTS dfs thadoopubuntu:/usr/Local/hadoop；bln$ 4/hdf5 dfsM f f m U U h I DE+1 = ,I H* N / 1 flj H, C" 11 lir&

6、#39;i. Problems 侬 Javadoc & Declaration S Console 曷vtermiidl已a HOF5Api Java Application/us r/lib/jvrii/jd?d-7-opeiijdk-dmd54/biii/jdva (Det 15r 20 log4j：WARN Please initiali2e the loq4j systen properly.Iog4j：WARN See http：//Log4j/l.2/taq.htnl#noconfig tor tore Inf /home/hadoop/

7、text.tit 已存在.将堇新至名为：/hcme/hmdQnp/tajct.txt GK时元康-(3) 将HDFS中指定文件的内容输出到终端中；m口叩iibuntu "UEr/losl/hmdoap/binS . /hdfs dfs -cat text i txt hello hadoop也 Problems Javadoc 忘 Declaration 曰 Console 居<terminated> HDFSApi Java Application /usr/lib/jvm/java-7-cperijdk-arTid64/bin/java (Dec 15, 2016 4

8、:5S: Loq4j：WXRH PLease initialize the Log4js/stes properly.漫取文件：/user/hadoop/text.txtL的4jWARN See http:/L/Log4j/1.2/1aq.html#nDCcnfig for more info.hello hadccip漫取完戌(4) 显示HDFSf指定的文件的读写权限、大小、创建时间、路径等信息;hadoopubuntu: / usryiocal/hidoop/bln ./hdfs df s -Is - h text, t>tt-rw-r- -r- -

9、1 hadoop supergroup13 2010-12-15 04:54 text,txtlcg4j：WARN Na apaenders ccu Ld be tcund for logger (arg.apache.hadcap.corf.Contigurat ion. iLcg4：WARN PLease initialize the lcg&j 5/ttem properly.log4j：WARN See http；/loggingapache,Qrg/log4/1,2/faq-htmlinoconfig for more info, 幽称：hdfs：/lccalhcst：909

10、&<user/hadoop/text,txt机不：rv-r-r -大小；13时回：261 12 15 G4:51:37谦职完应(5) 给定HDF汕某一个目录，输出该目录下的所有文件的读写权限、大小、创建时 间、路径等信息，如果该文件是目录，则递归输出该目录下所有文件相关信息；Rhadoopubuntu; /usr/local/hadoop/btn »/hdf s df s - Is，口,h f user/hridoop !-rw*r r*-1 hadoop supergroup13 2016-12*lb /user/hadoop/text,tIxt<termina

11、ted> HDFSApi Java A p p li catio n /u s r/li b/jvm/ja va-7-o penjd k-am d 64/bi n/j a va (Dec 15r 2016 5:02:40 AW 10用j :WARN No appendecould be Found fcr logger (arg.apache .hadoap.conf .Carf igUrati.Dn.depn Icgj ；WARN Please initialize th? Iog4j sister pro()erly.第WARN Sts http：/LafgingT4|pAche,o

12、r9/lo94j/lr2faq,htnl#noconfig for wore Info.1遢归)迂町目录f所为文判信息：/user/hadoopSSI? ； hdts ： /localhost ： 30GCAJser/hadoop/teKt .txt板阻：rw-r-r -K小：13时间:2016 12 IS 64:14:37慎敬亮通(6) 提供一个HDFS内的文件的路径，对该文件进行创建和删除操作。如果文件所在 目录不存在，则自动创建目录；h<kdgpubuntu:/u”八osl，hddo0p/btn5 ./hdfs dfs -m dirl/dirZ/filename |rn: '

13、;di.rl/dir2/fil.erarie,: No such file or dtrcctoryProblems 曲白川如亡国 ueclaration Q Lsnsole 理<terrrijnated''HDF5Api Java Applkation/usr/lib/jvm/java'7-openjdk'amd6-1/t injava (Dec 15,2016 5:Q6;21 lcg4jiWARN Md wpetiders could be found for ledger (qrg.apache.hddoop.conf.Confiyurationrc

14、 log4j：warm Please initial lie the log4j systen properly.Iog4j：WARM See http：/logging.apache,org/lcg4j/l.2/faq.htnl#ricconfig for nore info.创建文件史；/JS«r/hadc<p7inputMTtfflt": /user/hadoop/lnput/text.t> t(7) 提供一个HDFS的目录的路径，对该目录进行创建和删除操作。创建目录时，如 果目录文件所在目录不存在则自动创建相应目录；删除目录时，由用户指定当 该目录不为空

15、时是否还删除该目录；hadocpubuntu:/usr/lucal/hadoup/btn$ */hdfs dCs -nkdLr -p dLrl/dlr2 hadcopubuntu:/usr/locsl/hdocpybtn$ 4/hdfs dfs rndtr diri/dlr2 hadcapubuntu：/u£r/local/hadoop/binS t /hdfS fits -E -口 dtrl/dlrZh; "dl1/dlr2' ； Na &uch fll。 qdl±二t.gvPrebkms 麻 Javadoc心 Declaration日 Coni

16、olc S3<term nated>HDFSApi Java Application) /usr/lib/jvm/java-7-openjdk-amd64/bin/java (Dec 15, Z016 5Loq4j:H4版 no appenders could be found for logger (org.apache.hadoop.conf,corflratlog4j；WARN Please initialize the log4 s/stem procerly, Log4j:WARN See /log4j/1,2/faq.ht

17、inlineconiig for more info. 目录示为空,事除:/use r/ha doop/input(8) 向HDFS中指定的文件追加内容，由用户指定内容追加到原有文件的开头或结尾;h»doopAubuntu:/«sr/lacdl/hadoop/bln$ ,/hdfs dfs -appendToFtle local,txt textTtxtippendToFlle: /usr/lxc白Uhmdoop/bL。/LouLtxt (No such file or dtrectary)hadoap9(jbuntii：/usr/lacdl.i/hd(luup/bLri$

18、 ,/hdfs dfs -get text, t>tget: text.txt1: FtLe existshddoop0ubiintu;/usr/LciCdL/hddoup/bLfi$ ,/cat ttxt t txt » local. txtbah: ./cat: no such ftle or directoryhadoaMubiMtu;/usr/locaiyhadoop/bin$ ./hdfs dfs -copyFronLocal -f text.txt text.txtProblems Javsdoc I?. Perlaration Console 氯<termi

19、cated* HDFSApiJava Application/iJ£r/lib/jvm/java-7-opendk amdti/bin/java (DecJ 5r 2016 Log4j：WARN Nn appnder could b电 found for logger (org.apacha.hadoop.conf.Configura 1044 ：WAR.l Please Inltidlize the log4j system prouerly.Iog4j：WARN £ee http：/logging.apache.ong/log4j/I.5/faq.html+noconf

20、ig f&r more info. 已追加内容到文件末尾NEELhadoop/tBKt .txt(9) 删除HDFSH旨定的文件;Ihad»opubiintui/usr/l.Gcal/hodoop/t)in5 */hdfs dfs -rn text»txt 16/12/15 5:14:5L INFO fs.TrashPolicyDefault: Hanenode trash configurationi Dele tlon interval. = o minutes Efipller Interval = e nlnuces.Deleted text.t>t由

21、于已经删除，所以会显示操作失败。<cerminacea> nui3Api|ava AppitatQnj/u5r7UDvm"nY1openj(K=jmu64/DirvjHYa 13r 次1口 lcg4j：WARN No appendsrs could be found for logger (org.apache.hadoop.conf.Configure Lcg4j；wakn Please initialize the ：og4j system properly.Iag4j:WARN See http；//log4j/1.2/faq.h

22、tnlrfnoconfig f&r moinfo. 麋作矢雅(文件不存在或偌关般j(10)删除HDFS指定的目录，由用户指定目录中如果存在文件时是否删除目录; 由于前面已经删除，所以显示无该文件.hdcopubuntu:/usr/lccdt/hdoop/btnS /hdfs dfs -rmdtr d<rl/dirJ mdir: 'dtriydtr?': no such rile or directoryh4daopubuntu:/ur/locol/hdoop/binS /hdfs dfs -rn -R dirl/dir2 眄： dli/dirN'： N。

23、such file d die£tQv'* Problems 但 Javadoc i? Dechretion 曰 Console 贷(tErmihatEdHDFSApi dvd AppIkatHn /LKr/li b/j vm/j a vd-7-o pefjdkaind 64/bi n/j a va (Dec 15,2O1G 41M Log4j：WAftN No appenders could be found for logger (org,apactie.hadoop.conf.ConfiguraTion, laglj ： WARM Pl?asp i nitigll th

24、e 1 og4j systpri prnparly,l,ag4j Sep http： /Ingging, apache nrg/1 og4j/l .7/1aq ,htirlJSnoconTig for more Info.f录不.为免r不刑除“(11)在HDFS中，将文件从源路径移动到目的路径。hadoopubuntat/usf/Iac«I/hadoop/btn$ ,/hdfs dfs -rv text,txt text2«txt移动成功：berminatetf* HDFSApi Jav* 禺ppCcrt2n，uti7Ub/jwMdvaKopenjtft*ind64/bii

25、Vjdu(Dec 15, 2015 525:11 AMjl的4 ； warn No appenders could be fourvi for logger (org« apactie. hadoop. conf. con figuration .derecationl,log*j；w郴 Please initialize the logqj stet properly,1 ng：j：warn see hTTp：/IogglngFchcorg/log4j/i.2/faqhtiltnoconfig for »or Inf。将文岸 hdff ： /ucr/hsdDQp/XeKt

26、f txt 播劭界 hdfs ： /uwr/hadnop/new. txt2 .编程实现一个类"MyFSDataInputStream ”,该类继承“org.apache.hadoop.fs.FSData Inputstream : 要求如下：实现按行读取HDFS43指定文件的方法“readLine(): 如果读到文件末尾，则返回空，否则返回文件一行的文本。进行文档读取如下：；*_ Problems Javadoc IL. Declaration Q Console S3«emina【E6 MvFSDacainpjrstreaT Java Applicet on /us r/li a； j vm； j 3 va -7-o p en d k am d C4/b i n/j e va log4j;WARN No appenders could be found far logger