数据库教学课件：Hbase and the performance of accessing to HDFS(林俊宏)

HbaseTheHBaseHBaseisadistributedcolumn-orienteddatabasebuiltontopofHDFS.EasytoscaletodemandHBaseistheHadoopapplicationtousewhenyourequirereal-timeread/writerandom-accesstoverylargedatasets.UseMapReducetosearchHBasedependsonZooKeeperandbydefaultitmanagesaZooKeeperinstanceastheauthorityonclusterstate.DataModelAdatamodelsimilartoBigtable.adatarowhasasortablerowkeyandanarbitrarynumberofcolumnsthetableisstoredsparsely,rowsinthesametablecanhavewidelyvaryingnumbersofcolumnsConceptualViewPhysicalStorageViewExampleCapturenetworkpacketsintoHDFS,savetoafileforeveryminute.RunMapReduceapp,estimateflowstatus.counttcp,udp,icmppacketnumbercomputetcp,udp,orallpacketflowTheresultsavetoHBase.rowkeyandtimestamparethecaptruetimeRow-keyTimestampTcp:countTcp:flowUdp:countUdp:flow…2001003291011126985290000242343279890109273878978991645466…2010032910121269852960002899787109399930094812418163769889……………………DisplaySpecifystarttimeandstoptimetoscantablethenestimatedataanddisplayasflowgraph.SampleoutputTheperformanceofaccessingfilestoHDFSdirectlyandthroughaHDFS-basedFTPserverssh登入namenode下達指令上傳檔案至HDFS：hadoopfs-Ddfs.block.size=資料區塊位元組數-Ddfs.replication=資料區塊複製數量-put本機資料HDFS檔案目錄由HDFS下載檔案：hadoopfs-getHDFS上的資料本機目錄AccessingfilestoHDFSdirectly

(1/7)觀察透過HDFS參數的調整，讓HDFS在不同條件下的檔案讀取效能。之後的標題中若標示R=1，表示某檔案在HDFS中的複製（備份）數量。AccessingfilestoHDFSdirectly

(2/7)AccessingfilestoHDFSdirectly

(3/7,R=1)

MB級(100MB)GB級(3.3GB)1M4120.52M3.7103.24M3.493.58M3.281.716M2.784.832M2.565.964M1.865.1128M1.951.6256M1.962.7512M1.977（橫軸表示資料分割區塊大小，單位：byte）（縱軸表示一份資料完全寫入HDFS所需要的時間，單位：秒）AccessingfilestoHDFSdirectly

(4/7,R=1)

MB級(100MB)GB級(3.3GB)1M2.361.52M2.2634M2.162.48M2.161.816M2.161.732M2.162.264M2.160.1128M261.8256M263.1512M262.1（橫軸表示資料分割區塊大小，單位：byte）（縱軸表示一份資料完全從HDFS讀出所需要的時間，單位：秒）AccessingfilestoHDFSdirectly

(5/7,R=2)

MB級(100MB)GB級(3.3GB)1M3.8224.52M3.4190.14M3.21478M3131.216M2.8133.532M3.1124.964M3.2118.7128M3.2120.5256M3.3143.3512M3.3124.9（橫軸表示資料分割區塊大小，單位：byte）（縱軸表示一份資料完全寫入HDFS所需要的時間，單位：秒）AccessingfilestoHDFSdirectly

(6/7,R=2)

MB級(100MB)GB級(3.3GB)1M2.363.42M2.361.54M2.261.58M2.160.116M26032M2.158.464M1.958128M2.161.7256M261.5512M258.5（橫軸表示資料分割區塊大小，單位：byte）（縱軸表示一份資料完全從HDFS讀出所需要的時間，單位：秒）結論在運行NameNodedaemon的namenodeserver上直接上下載檔案，原則上資料區塊大小以64MB或128MB效能較佳。資料區塊複製數越多，雖在檔案寫入時會花較久的時間，但在檔案讀取時速度會些許提升。AccessingfilestoHDFSdirectly

(7/7)使用者用FTPclient連上FTPserver後lfs表示一般的FTPserverdaemon直接存取localfilesystem。HDFS表示由我們撰寫的FTPserverdaemon，透過與位在同一台server上的NameNodedaemon溝通後，存取HDFS。之後上傳／下載完檔案花費之總秒數皆為測量3次秒數平均後之結果網路頻寬約維持在10Mb/s~12Mb/s間AccessingfilesthroughaHDFS-basedFTP

server(1/3)AccessingfilesthroughaHDFS-basedFTPserver(2/3)

lfsHDFS0.5GB46.3362.331.0GB95138.671.5GB141.671992.0GB188.672702.5GB2373463.0GB288.334003.5GB3454714.0GB383.67472（橫軸：上傳單一檔案GB數）（縱軸：上傳完檔案花費總秒數）（HDFS：檔案區塊大小128MB，複製數=2）AccessingfilesthroughaHDFS-basedFTPserver(3/3)

lfsHDFS0.5GB48451.0GB92911.5GB141137.332.0GB192185.672.5GB236.67226.333.0GB273.332783.5GB322320.334.0GB380.33378.67（橫軸：下載單一檔案GB數）（縱軸：下載完檔案花費總秒數）（HDFS：檔案區塊大小128MB，複製數=2）Hadoop認證分析thenamenodehasnonotionoftheidentityofthereal

user。（沒有真實用戶的概念）UserIdentity：Theusernameistheequivalentof「whoami」.Thegrouplististheequivalentof「bash-cgroups」.Thesuper-useristheuserwiththesameidentityasnamenodeprocessitself.Ifyoustartedthenamenode,thenyouarethesuper-user.WhyUsingProxyto

connectnamenodeDataNodesdonotenforceanyaccesscontrolonaccessestoitsdatablocks。(client可與datanode直接連線，提供BlockID即可read、write)。Hadoop

client(anyuser)canaccessHDFSorsubmitMapreduceJob。HadooponlyworkswithSOCKSv5.(inclient，ClientProtocolandSubmissionProtocol

)結論：hadoop(PrivateIP叢集)+RADIUS+SOCKSproxy。結構結構HadoopSOCKS只需在Hadoopclient設定SOCKS連線，Namenode無需設定。User認證使用SOCKSprotocol的method(username、password)辨識Proxy

transfer的權限。由RADIUS

Server紀錄user是否可以存取hadoop。(user-group)