技术学习adtec-hadoop培训第二周

1、HadoopHadoop【】【】Raid3Raid34556各各7数据块的8数据块的899建立pipeline依次将目标数据块写入各个datanode建立pipeline依次将目标数据块写入各个datanode可以在hdfs-site.xmlDatanodehdfs可以在hdfs-site.xmlDatanodehdfs#-*-coding:UTF-8-*- import sys #-*-coding:UTF-8-*- import sys if name = main /没有按时发送心跳的datanodeI/O如果datanodenamenode会检测没有按时发送心跳的datanodeI/O

2、如果datanodenamenode会检测Numberofminutesbet Numberofminutesbet ns.Ifzero, thetrash feature 在Hadoop2.x在Hadoop2.x注意，hadoop没有当前目录的概念，也没有cd注意，hadoop没有当前目录的概念，也没有cd上传文件到上传文件到设置sshdatanode和tasktracker（hadoop-daemon.sh设置sshdatanode和tasktracker（hadoop-daemon.shstart-all.shstart-all.shHDFSAPIHDFSAPIpublicRLCatsi

3、c publicicpublicRLCatsic publicicvoidmain(String args)throwsExceptionInputStreamin=try in = new URL(args0).openStream(); IOUtils.copyBytes(in,System.out,4096,false); finally 使到运行javac，jps使到运行javac，jps添加Java和Hadoopimport import添加Java和Hadoopimport import 编译时指定编译时指定build.xml propertyname=buildbuild.xml

4、 propertyname=buildlocation=buildproperty filesetincludename=*/*.jarfilesetincludename=*/*.jar续 javacincludeantruntime=falsesrcdir=srcclasspath refid=hadoop-jarjarfile=续 javacincludeantruntime=falsesrcdir=srcclasspath refid=hadoop-property name=classpath refid=hadoop-用ant用antpublicclassemo publicpub

5、licclassemo publicicvoidmain(Stringargs)throwsIOExceptionConfigurationconf=newFileSystemfs= Pathfile=续if(fs.exists(file)else/Writing toln(File 续if(fs.exists(file)else/Writing toln(File /Readingfrome toHDFSJava Stringdata=指南第3章“Theerface指南第3章“Theerface安装yum-y installgccgcc-c+autoconf安装yum-y installgc

6、cgcc-c+autoconf 测试HDFSCAPI：一段C测试HDFSCAPI：一段C/Followingis a libhdfsleadaptedfromthe src/c+/libhdfs/hdfs_write.cof the/Followingis a libhdfsleadaptedfromthe src/c+/libhdfs/hdfs_write.cof the Hadoopargc,char *argv)hdfsFSfs =if (!fs)f(stderr,Cannotconnectto char* fileName=char* message= size= eto HDFSC续

7、exists =hdfsExists(fs,if(exists-1)f(stdout,续exists =hdfsExists(fs,if(exists-1)f(stdout,File %sexists!n,/ Createand openfile forhdfsFile outFile = hdfsOpenFile(fs,fileName,O_WRONLY|O_CREAT,0, 0, if(!outFile)f(stderr,Failedto open%sforwriting!n,/writetohdfsWrite(fs,outFile,(void*)message,hdfsCloseFile

8、(fs,续/Openfile forhdfsFileinFile =hdfsOpenFile(fs,fileName,O_RDONLY,续/Openfile forhdfsFileinFile =hdfsOpenFile(fs,fileName,O_RDONLY,0, 0, if(!inFile)f(stderr,Failed to open%s for reading!n,char*data= malloc(sizeof(char)*/ Read fromtSize readSize= hdfsRead(fs,inFile, (void*)data,f(stdout,%sn,hdfsClos

9、eFile(fs,return 0;gcchdfs_cpp_demo.c-I$HADOOP_HOME/src/c+/libhdfsgcchdfs_cpp_demo.c-I$HADOOP_HOME/src/c+/libhdfs -L$HADOOP_HOME/c+/Linux-amd64-64/lib/-lhdfs-L$JAVA_HOME/jre/lib/amd64/server-ljvm-o利用ant利用ant64/lib:$JAVA_HOME/jre/lib/amd64/server./hdfs_cpp_demo64/lib:$JAVA_HOME/jre/lib/amd64/server./h

10、dfs_cpp_demoHadoop2.xHDFS Hadoop2.xHDFS BlockStorageService（1namenode服务）提供datanodeBlockStorageService（1namenode服务）提供datanode2：datanode1.xHDFS制在2.x1.xHDFS制在2.x多namenodes及datanodedatanode多namenodes及datanodedatanode块池（Block每一个datanode为所有的block 块池（Block每一个datanode为所有的block BlockIDblock一个Namenode datanod

11、e为其他Namenode1.x的Namenode2.xnamenode1.x的Namenode2.xnamenode Hash返一在分布式系统里常用采用ntSideMountHash返一在分布式系统里常用采用ntSideMountTablehdfs-c-hdfs-c-续nn-host2:rpc-续nn-host2:rpc-. Other commonconfigurationStep1clusterIdamenode -format -clusterId StepStep1clusterIdamenode -format -clusterId Step2clusterIdamenode-format -clusterIdhdfsdfsadmin-allowSnapshothdfsdfsadmin