HDFS块检查命令Fsck机理的分析

HDFS块检查命令Fsck机理的分析前⾔在HDFS中,所有的⽂件都是以block块的概念⽽存在的,那么在这样海量的⽂件数据的情况下,难免会发⽣⼀些⽂件块损坏的现象,那么有什么好的办法去发现呢.答案是使⽤HDFS的fsck相关的命令.这个命令独⽴于dfsadmin的命令,可能会让部分⼈不知道HDFS中还存在这样的命令,本⽂就来深度挖掘⼀下这个命令的特殊的⽤处和内在机理的实现.Fsck命令其实说到fsck命令本⾝,熟悉Linux操作系统的⼈,可能或多或少听到过或使⽤过这个命令.Fsck命令的全称为filesystemcheck,更加类似的是⼀种修复命令.当然,本⽂不会讲⼤量的关于操作系统的fsck怎么⽤,⽽是HDFS下的fsck的使⽤,在bin/hdfsfsck下还是有很多可选参数的.Fsck参数使⽤本⼈在测试集群中输⼊hdfsfsck命令,获取了帮助信息,在此信息中展⽰了最全的参数使⽤说明:$hdfsfsckUsage:hdfsfsck<path>[-list-corruptfileblocks|[-move|-delete|-openforwrite][-files[-blocks[-locations|-racks]]]]<path>startcheckingfromthispath-movemovecorruptedfilesto/lost+found-deletedeletecorruptedfiles-filesprintoutfilesbeingchecked-openforwriteprintoutfilesopenedforwrite-includeSnapshotsincludesnapshotdataifthegivenpathindicatesasnapshottabledirectoryortherearesnapshottabledirectoriesunderit-list-corruptfileblocksprintoutlistofmissingblocksandfilestheybelongto-blocksprintoutblockreport-locationsprintoutlocationsforeveryblock-racksprintoutnetworktopologyfordata-nodelocations-storagepoliciesprintoutstoragepolicysummaryfortheblocks-blockIdprintoutwhichfilethisblockIdbelongsto,locations(nodes,racks)ofthisblock,andotherdiagnosticsinfo(underreplicated,corruptedornot,etc)简单的总结⼀下,⾸先是必填参数和命令名:bin/hdfsfsck<path>然后是⼀堆的可选参数:-move:移动损坏的⽂件到/lost+found⽬录下-delete:删除损坏的⽂件-files:输出正在被检测的⽂件-openforwrite:输出检测中的正在被写的⽂件-includeSnapshots:检测的⽂件包括系统snapShot快照⽬录下的-list-corruptfileblocks:输出损坏的块及其所属的⽂件-blocks:输出block的详细报告-locations:输出block的位置信息-racks:输出block的⽹络拓扑结构信息-storagepolicies:输出block的存储策略信息-blockId:输出指定blockId所属块的状况,位置等信息具体参数功能对应到相应的程序会在下⽂的分析中进⾏详细的阐述.Fsck过程调⽤Fsck过程的调⽤指的是从终端机器输⼊到最终fsck在HDFS内部被执⾏的整个过程.中间穿过的类的其实不多,本⼈做了⼀张简图:上图的调⽤形式,可以说是三层调⽤的结构.DFSck就是暴露在最外层的类.我们再来规整规整中间的过程.输⼊fsck直接调⽤到的就是此类.DFSck内部会发送http请求的⽅式,根据参数构造URL请求地址,发送到下⼀个处理对象中.下⼀个处理对象就是FsckServlet.FsckServlet在这⾥相当于⼀个过渡者,马上调⽤真正操作类NamenodeFsck.NamenodeFsck在这⾥会取出请求参数,然后在HDFS内部做真正的fsck检测操作.Fsck原理分析Fsck原理分析将会展⽰更加细致的fsck过程调⽤.按照上⼩节的提到的3层调⽤,同样我们也分为3个层次的渐近性的分析.DFSck请求构造你可以把此类想象成DFSAdmin.⾸先进⼊命令输⼊处理⼊⼝⽅法:publicintrun(finalString[]args)throwsIOException{if(args.length==0){printUsage(System.err);return-1;}try{returnUserGroupInformation.getCurrentUser().doAs(newPrivilegedExceptionAction<Integer>(){@OverridepublicIntegerrun()throwsException{returndoWork(args);}});}catch(InterruptedExceptione){thrownewIOException(e);}}在doWork⽅法中,马上就看到了对于参数的判别分类,同时开始构造不同的参数请求.privateintdoWork(finalString[]args)throwsIOException{finalStringBuilderurl=newStringBuilder();url.append("/fsck?ugi=").append(ugi.getShortUserName());Stringdir=null;booleandoListCorruptFileBlocks=false;for(intidx=0;idx<args.length;idx++){if(args[idx].equals("-move")){url.append("&move=1");}elseif(args[idx].equals("-delete")){url.append("&delete=1");}elseif(args[idx].equals("-files")){url.append("&files=1");}elseif(args[idx].equals("-openforwrite")){url.append("&openforwrite=1");}elseif(args[idx].equals("-blocks")){url.append("&blocks=1");}elseif(args[idx].equals("-locations")){url.append("&locations=1");}elseif(args[idx].equals("-racks")){url.append("&racks=1");}elseif(args[idx].equals("-storagepolicies")){url.append("&storagepolicies=1");}...不同类型的参数后⾯接的参数值也不⼀定相同,⽐如-blockId后⾯则会跟连续的blockId....}elseif(args[idx].equals("-blockId")){StringBuildersb=newStringBuilder();idx++;while(idx<args.length&&!args[idx].startsWith("-")){sb.append(args[idx]);sb.append("");idx++;}url.append("&blockId=").append(URLEncoder.encode(sb.toString(),"UTF-8"));...请求url构造好之后,就会发起请求URLpath=newURL(url.toString());URLConnectionconnection;try{connection=connectionFactory.openConnection(path,isSpnegoEnabled);}catch(AuthenticationExceptione){thrownewIOException(e);}然后获取响应回复,直接输出到终端上.InputStreamstream=connection.getInputStream();BufferedReaderinput=newBufferedReader(newInputStreamReader(stream,"UTF-8"));Stringline=null;StringlastLine=null;interrCode=-1;try{while((line=input.readLine())!=null){out.println(line);lastLine=line;}}finally{input.close();}OK,DFSck最外层⾯的调⽤过就⾛通了.FsckServlet请求处理上个步骤中url请求会转到FsckServlet中处理,类似代理⼈的⾓⾊,然后马上调⽤NamenodeFsck进⾏处理/**Handlefsckrequest*/@OverridepublicvoiddoGet(HttpServletRequestrequest,HttpServletResponseresponse)throwsIOException{@SuppressWarnings("unchecked")finalMap<String,String[]>pmap=request.getParameterMap();...finalUserGroupInformationugi=getUGI(request,conf);try{ugi.doAs(newPrivilegedExceptionAction<Object>(){@OverridepublicObjectrun()throwsException{NameNodenn=NameNodeHttpServer.getNameNodeFromContext(context);finalFSNamesystemnamesystem=nn.getNamesystem();finalBlockManagerbm=namesystem.getBlockManager();finalinttotalDatanodes=namesystem.getNumberOfDatanodes(DatanodeReportType.LIVE);newNamenodeFsck(conf,nn,bm.getDatanodeManager().getNetworkTopology(),pmap,out,totalDatanodes,remoteAddress).fsck();returnnull;}});}catch(InterruptedExceptione){response.sendError(400,e.getMessage());}}NamenodeFsck的fsck处理上节中最后⼀个步骤最终调⽤的就是NamenodeFsck的fsck⽅法.在进⼊这个⽅法之前,先看⼀下,这个类的⼀些关键变量.privateStringlostFound=null;privatebooleanlfInited=false;privatebooleanlfInitedOk=false;privatebooleanshowFiles=false;privatebooleanshowOpenFiles=false;privatebooleanshowBlocks=false;privatebooleanshowLocations=false;privatebooleanshowRacks=false;privatebooleanshowStoragePolcies=false;privatebooleanshowCorruptFileBlocks=false;这些布尔类型的变量对应的就是控制fsck帮助信息所展⽰的各个参数.个⼈感觉fsck⽅法内部的处理顺序看起来有点乱,为了便于⼤家的理解,这⾥对指定参数进⾏指定分析的⽅式,就不转⾏对照的描述了.-list-corruptfileblocks第⼀个参数⽅法-list-corruptfileblocks,展⽰丢失/损坏的块.if(showCorruptFileBlocks){listCorruptFileBlocks();return;}然后调⽤到同名⽅法listCorruptFileBlocks.privatevoidlistCorruptFileBlocks()throwsIOException{Collection<FSNamesystem.CorruptFileBlockInfo>corruptFiles=namenode.getNamesystem().listCorruptFileBlocks(path,currentCookie);intnumCorruptFiles=corruptFiles.size();...out.println("Cookie:\t"+currentCookie[0]);for(FSNamesystem.CorruptFileBlockInfoc:corruptFiles){out.println(c.toString());}out.println("\n\nThefilesystemunderpath'"+path+"'has"+filler+"CORRUPTfiles");out.println();}此⽅法最终会调⽤到FSNamesystem的listCorruptFileBlocks⽅法,注意这⾥还传⼊了⼀个特别的参数currentCookie.这个参数的作⽤可是⾮常的巧妙的.进⼊FSNamesystem的⽅法,⾸先初始化对象损坏⽂件块对象:ArrayList<CorruptFileBlockInfo>corruptFiles=newArrayList<CorruptFileBlockInfo>();⽅法返回的对象也即是此对象.然后进⼊关键的损坏⽂件的判断逻辑//Doaquickcheckifthereareanycorruptfileswithouttakingthelockif(blockManager.getMissingBlocksCount()==0){if(cookieTab[0]==null){cookieTab[0]=String.valueOf(getIntCookie(cookieTab[0]));}if(LOG.isDebugEnabled()){LOG.debug("therearenocorruptfileblocks.");}returncorruptFiles;}blockManager的getMissingBlocksCount⽅法取的就是损坏块队列的⼤⼩.publiclonggetMissingBlocksCount(){//notlockingreturnthis.neededReplications.getCorruptBlockSize();}如果此⽅法的Count返回值有值,就是⼤于0,则⽅法执⾏继续//获取损坏块的block迭代器finalIterator<Block>blkIterator=blockManager.getCorruptReplicaBlockIterator();//取出cookie值作为标记位,跳过标记下标之前的⽂件,代表已经浏览过intskip=getIntCookie(cookieTab[0]);for(inti=0;i<skip&&blkIterator.hasNext();i++){blkIterator.next();}while(blkIterator.hasNext()){Blockblk=blkIterator.next();finalINodeinode=(INode)blockManager.getBlockCollection(blk);//更新skip跳过值skip++;if(inode!=null&&blockManager.countNodes(blk).liveReplicas()==0){Stringsrc=FSDirectory.getFullPathName(inode);if(src.startsWith(path)){corruptFiles.add(newCorruptFileBlockInfo(src,blk));count++;if(count>=DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED)break;}}}//更新cookie标记值cookieTab[0]=String.valueOf(skip);cookie的作⽤就是如上注释所说,获取到此返回损坏⽂件列表后,会在上⼀⽅法中将结果输出for(FSNamesystem.CorruptFileBlockInfoc:corruptFiles){out.println(c.toString());}fsck-path默认处理⽅法fsck的默认处理⽅法指的就是fsck+path的⽅法,为什么紧接着讲这个⽅法呢,因为fsck的path⽅法处理也包括了扫描损坏块的⽅法,但是在逻辑上与-list-corruptfiles竟然还不⼀样,这⼀点本⼈在阅读的时候,也是感到⽐较吃惊的.⾸先⼤家传⼊的path会被传⼊到内部⽅法check中处理Resultres=newResult(conf);check(path,file,res);out.println(res);out.println("Numberofdata-nodes:\t\t"+totalDatanodes);out.println("Numberofracks:\t\t"+networktopology.getNumOfRacks());然后会进⾏⽬录,⽂件的判断,如果是⽬录,则进⾏递归调⽤if(file.isDir()){//如果快照⽬录包含此路径,则递归快照⽬录下的pathif(snapshottableDirs!=null&&snapshottableDirs.contains(path)){StringsnapshotPath=(path.endsWith(Path.SEPARATOR)?path:path+Path.SEPARATOR)+HdfsConstants.DOT_SNAPSHOT_DIR;HdfsFileStatussnapshotFileInfo=namenode.getRpcServer().getFileInfo(snapshotPath);check(snapshotPath,snapshotFileInfo,res);}...do{assertlastReturnedName!=null;thisListing=namenode.getRpcServer().getListing(path,lastReturnedName,false);if(thisListing==null){return;}HdfsFileStatus[]files=thisListing.getPartialListing();//递归变量此path的⼦⽂件,如果此path是⽬录的话for(inti=0;i<files.length;i++){check(path,files[i],res);}lastReturnedName=thisListing.getLastName();}while(thisListing.hasMore());return;}在接下来的分析检测⽂件时,会进⾏相应指标的统计值更新isOpen=blocks.isUnderConstruction();if(isOpen&&!showOpenFiles){//Wecollectthesestatsaboutopenfilestoreportwithdefaultoptionsres.totalOpenFilesSize+=fileLen;res.totalOpenFilesBlocks+=blocks.locatedBlockCount();res.totalOpenFiles++;return;}res.totalFiles++;res.totalSize+=fileLen;res.totalBlocks+=blocks.locatedBlockCount();下⾯是关键的判断path下所属的block块中的损坏块的判断逻辑:...for(LocatedBlocklBlk:blocks.getLocatedBlocks()){ExtendedBlockblock=lBlk.getBlock();booleanisCorrupt=lBlk.isCorrupt();StringblkName=block.toString();...这⾥直接利⽤了LocatedBlock内部的isCorrupt的⽅法,然后进⾏corrupt计数累加//CheckifblockisCorruptif(isCorrupt){corrupt++;res.corruptBlocks++;out.print("\n"+path+":CORRUPTblockpool"+block.getBlockPoolId()+"block"+block.getBlockName()+"\n");}⽽且在这⾥,missing块的判断逻辑是独⽴于corrupt块的.//重新进⾏块副本数的统计NumberReplicasnumberReplicas=namenode.getNamesystem().getBlockManager().countNodes(block.getLocalBlock());//获取存在的副本数intliveReplicas=numberReplicas.liveReplicas();//如果当前副本数确实为0,则表明已经是missing块if(liveReplicas==0){report.append("MISSING!");res.addMissing(block.toString(),block.getNumBytes());missing++;missize+=block.getNumBytes();}else{重新回顾以上check⽅法中的这2类块判断逻辑,第⼆个missing块的判断逻辑,我个⼈认为是没有问题的,但是第⼀个corrupt的判断我个⼈感觉可能有点问题,尽管说LocatedBlock提供了内部⽅法isCorrupt,但是我在查询isCorrupt的调⽤处时发现绝⼤多数情况下都是false参数默认传⼊的,⽽且在数据实时性和有效性上⽽⾔,这个⽅法是没有-list-corruptfiles参数来的快与准的(个⼈观点,可能理解不同).因为-list-corruptfiles直接是从FSNamesystem类中取的,⼀⽅⾯代表的已经是最新的损坏数据情况了.fsck-delete/-move这2个命令作⽤是找到损坏块之后,打算要做什么事情,就是下⾯2⾏代码所控制的:...}else{if(doMove)copyBlocksToLostFound(parent,file,blocks);if(doDelete)deleteCorruptedFile(path);}...LostFound指的是/lost+found⽬录,下,就是说-move参数会将损坏块⽂件,移⾄此⽬录下,⽽-delet则会调⽤直接删除的⽅法privatevoiddeleteCorruptedFile(Stringpath){try{namenode.getRpcServer().delete(path,true);LOG.info("Fsck:deletedcorruptfile"+path);}catch(Exceptione){LOG.error("Fsck:errordeletingcorruptedfile"+path,e);internalError=true;}}其实这2个命令的还是⽐较有⽤的.如果集群中存在⼤量损坏块数据的情况时,如果不及时进⾏清理,会出现⼤量客户端读写操作的失败,因为元数据虽然存在,但是真实数据已经损坏,读写操作必然会抛出异常.fsck辅助显⽰参数以上⼏个是fsck的主要参数,下⾯是⼀些辅助的次要⼀些的参数.-locations/-racksif(showLocations||showRacks){StringBuildersb=newStringBuilder("[");for(intj=0;j<locs.length;j++){if(j>0){sb.append(",");}if(showRacks)sb.append(NodeBase.getPath(locs[j]));elsesb.append(locs[j]);}sb.append(']');report.append(""+sb.toString());}-storagepoliciesif(this.showStoragePolcies){storageTypeSummary=newStoragePolicySummary(namenode.getNamesystem().getBlockManager().getStoragePol