《HDFS简介详尽版》PPT课件.ppt

HDFS简介,杨林三2012.08.28,提纲,文件系统HDFS实验,文件系统概览,文件系统数据结构,文件系统数据结构,文件系统工作流程,LinuxExt2,LinuxExt3,NFS,小结,Hadoop,Hadoop,HDFS目标,Targets&CharactersVerylargefilesGB,TB,PB+Millionsoffiles10K+nodesStreamingdataaccessWrite-once,read-many-timesGooddatacoherencyTimetoreadwholedatasetmattersmorethantimetoseekCommodityhardwareCheapNodefailure:thechanceoverlargeclusterishighRedundancythroughreplicasFailurehandlingandrecovery,HDFS目标,Targets&CharactersOptimizedforbatchprocessingmovecomputationratherthandatalocationsofdataexposedtocomputationNotgoodforLow-latencydataaccessmsHighthroughputagainstLow-latencyLotsofsmallfilesMetadatamayrunoutofmemorySeektimeoverreadtimeMultiplewriters,arbitraryfilemodificationAsinglewriterWriteatendWrite-once,read-many-times,HDFS架构,Master/WorkersNameNodeMaster:onesingleFilenamespaceDirectoriesandfilesMetadataInmemoryMappingfilename-blocksblocks-datanodeClusterconfigurationReplicationManagement,DataNodesWorkers:manyAblockserverStoreblocksinlocalFSStoremetadataofblocksServedatatoclientsFacilitatepipelinetootherdatanodes,HDFS架构,DataStorage,Blocks/Chunks,Blocksize64/128/256MBConfigurablewhysobig?transferoverseekseek:10ms,transferrate:100MB,seektime/transfertime=1%=blocksize=100MBblocksofafiledistibutedoverdatanodesaspossibleasingleFILE,existingofNblocks,isstoredonMnodes,Namenode,Logofcreation,deletion,renameofthenamespace,NamespacetreeiscachedinRAMandstoredpermanentlyinFSImage,Editlogtakesrecodeofopen,close,renamefileordirectory,etc,SecondaryNamenode,Why?PeroidicallyMergeeditsandfsimagePreventeditlogfrombeingtoolargeProvidecheckpointofNamenodesfsimageasbackupincaseofNamenodecrash,NamenodeHigh-Availability,FsImageandEditLogarecentraldatastructuresofHDFS.AcorruptionofthesefilescancauseaHDFSinstancetobenon-functional.Forthisreason,aNamenodecanbeconfiguredtomaintainmultiplecopiesoftheFsImageandEditLog.MultiplecopiesoftheFsImageandEditLogfilesareupdatedsynchronously.Meta-dataisnotdata-intensive.,NamenodeSafemode,MergeFSImageandEditsEntersafemodeOffersread-onlyviewoffilesystemtoclientsBlock-locationinformationcollectionblock-locationinfonotinfsimagedatanodescheckintoreporttheirblockscheckblocksminimalreplicationcondition:99.9%blocksmeetminimumreplicationlevel(configurable,default1)exitsafemodeifminimalreplicationconditionreachesReplicateblocksifnecessaryblockswhoesreplicanumberislessthan3,HDFSoperations,MetadataoperationsCommunicatewithnamenodeonlyls,lsr,df,du,chmod,chown.R/W(block)operationsCommunicatewithnamenodeanddatanodesput,copyFromLocal,copyToLocal,tail.,DataWriting,DataReading,ReplicaPlacement,Criticaltoreliabilityandperformance.ReplicasareplacedReplicationfactoris31onanodeinalocalrack1onadifferentnodeinthelocalrack1onanodeinadifferentrack.Replicationfactor31/3ofthereplicaonanode1/3onondifferentnodeinthesamerack1/3distributedevenlyacrossremainingracks.ReplicaselectionThenearestoneforread,NetworkTopology,Characters3040nodes/rack1GB+switchforrackuplinktoacoreswitchorrouter1GB+aggregatebandwidthnodesonsamerackmuchhigherthandifferentracks,Bandwidth&DistancesProcessesonthesamenodeDifferentnodesonthesamerackNodesondifferentracksinthesamedatacenterNodesindifferentdatacenters,DatanodeFailure,DiskfailureDataNode正常服务坏掉的磁盘上的数据尽快通知NameNodeMachineDown问：NameNode怎么知道DataNode挂掉了？答：datanode每3秒钟向namenode发送心跳，如果10分钟datanode没有向namenode发送心跳，则namenode认为该datanode已经dead，namenode将取出该datanode上对应的block，对其进行复制。,HDFS压缩,实验-拷贝数据,hadoopfs-copyFromLocal/home/hduser0/hadoop/practice/data/ntes_logs/news_20120813.23.log.gz/data/input/ntes_logs/,实验-数据块,实验-LineCount,后续,LZO压缩格式实验日志合并原perl流程多台js服务器rsync合并机合并读2分钟的日志按时间排序url正则匹配：多个正则表达式Hadoop可能的方案js服务器可以作为Data