细细品味Hadoop-Hadoop集群（第12期）-HBase应用开发

——Hadoop集群(第12期)精华集锦csAxp虾皮工作室ogscomxiapi河北工业大学——软件工程与理论实验室编辑：虾皮1Hadoop集群(第12期)HBaseHBase为用户提供了一个非常方便的使用方式，我们称之为“HBaseShell”。HBaseShellHBaseHBaseShell地创建、删hbaseshell河北工业大学——软件工程与理论实验室编辑：虾皮2HBaseShell命令描述ee的值)all定行的所有元素e获取行或单元(cell)的值、行或列的值putols描来获取对应的值runcate表versionMultiplebindingswerefoundontheclasspath河北工业大学——软件工程与理论实验室编辑：虾皮3SLF4JAPIisdesingedtobindwithoneandonlyoneunderlyingloggingframeworkatatime.Ifmorethanonebindingispresentontheclasspath,SLF4Jwillemitawarning,listingthelocationofthosebindings.Whenthishappens,selecttheoneandonlyonebindingyouwishtouse,andremovetheotherbindings.Forexample,ifyouhavebothslf4j-simple-1.6.4.jarandslf4j-nop-1.6.4.jarontheclasspathandyouwishtousethenop(no-operation)binding,thenremoveslf4j-simple-1.6.4.jarfromtheclasspath.NoteThewarningemittedbySLF4Jisjustthat,awarning.SLF4Jwillstillbindwiththefirstframeworkitfindsontheclasspath.，待以后解决。api1aoxue2s。create‘scores’,’grad’,’course’河北工业大学——软件工程与理论实验室编辑：虾皮42)list命令3)describe命令describe‘scores’4)put命令。格式：put表名，行名，列名([列族:列名])，值put‘scores’,’xiapi’,’grad:’,’1’给“xiapi”这一行的数据的列族“course”添加一列“<china,97>”。put‘scores’,’xiapi’,’course:china’,’97’河北工业大学——软件工程与理论实验室编辑：虾皮5给“xiapi”这一行的数据的列族“course”添加一列“<math,128>”。put‘scores’,’xiapi’,’course:math,’128’给“xiapi”这一行的数据的列族“course”添加一列“<english,85>”。put‘scores’,’xiapi’,’course:english,’85’get命令scores”中的行“xiapi”的相关数据。get‘scores’,’xiapi’河北工业大学——软件工程与理论实验室编辑：虾皮6get‘scores’,’xiapi’,’course:math’getscores,’xiapi’,{COLUMN=>’course:math’}scan命令查看表“scores”中的所有数据。scan‘scores’scanscoresCOLUMNS=>‘course’}河北工业大学——软件工程与理论实验室编辑：虾皮7delete命令hdelete‘scores’,’xiaoxue’,’course:math’ss这里我们重点介绍与HBase数据库存储管理相关的内容，其所涉及的主要类包括：河北工业大学——软件工程与理论实验室编辑：虾皮8httphbaseapacheorg/apidocs/index.htmlJava类HBase数据模型eAdmin数据库(database)seConfiguration表(table)列族(columnfamily)列修饰符(columnqualifier)HBaseConfigurationchehadoophbaseHBaseConfiguration描描述oidddRescourePathfile所指的文件来添加资源oidar的属性ingnameBooleandefaultValueoidameStringvalue来设置值oideanStringnameBooleanvalueHBaseConfigurationhconfignewHBaseConfiguration();hbasezookeeperpropertyclientPortaseConfiguration2)HBaseAdmin类pachehadoophbaseclientHBaseAdmin河北工业大学——软件工程与理论实验室编辑：虾皮9描述oidumnStringtableNameHColumnDesciptorcolumnoidcheckHBaseAvailabeHBaseConfigurationconf)oidesc同步操作oidytetableName在的表oidetableName态oidtableName态oroidlebytetableNametorhtd修改表的模式，是异步的操作，可能需要oleanbleExistsStringtableNameHBaseAdminadminnewHBaseAdmin(config);etablenamecriptoreHTableDescriptorHTableDescriptor的列族。描描述oidFamilyHColumnDescriptorlumnDescriptorremoveFamily(byte[]column)byte[]Namebyte[]oidleDescriptortablenamehtdaddFamilynewHColumnDescriptor(“Family”));河北工业大学——软件工程与理论实验室编辑：虾皮104)HColumnDescriptor类r常在创建表或为表添加列族时候使用。列族被创建后不能直接修改，只能通过删除然后重or描描述byte[]Namebyte[]oidbleDescriptortablenameHColumnDescriptorcol=newHColumnDescriptor(“content:”);htd.addFamily(col);clientHTable描述oidytefamilybytequalifierbyte]value,Putput)eroidse源或挂起内部缓冲区的更新oleanstsGetgetsultetGetget元格所对应的值byte[][]EndKeys每一个区域的结束键值ultScannergetScannerbytefamily)rbyte[]ooleanisTableEnabled(HBaseConfigurationconfameoidput(Putput)河北工业大学——软件工程与理论实验室编辑：虾皮11BytestoBytestablenameResultScannerscannertable.getScanner(family);6)Put类chehadoophbaseclientPut于对单个行执行添加操作。描述oidybytequalifierbytevalueilybytequalifierlongtsbytevaluebyte[]RowwLocktRowLockleansEmptyngtimestampBytestoBytestablenamePutp=newPut(brow);//为指定行(brow)创建一个Put操作ifiervaluetableput(p);7)Put类achehadoophbaseclientGet。描述nbytefamilybytequalifier符对应的列addFamilybytefamily获取其对应的所有列gminStamplongmaxStamp本号ilterFilterfilterBytestoBytestablename河北工业大学——软件工程与理论实验室编辑：虾皮12GetgnewGetBytestoBytesrow);8)Result类pachehadoophbaseclientResultn描描述oleannbytefamilybyte[]qualifier)否存在NavigableMap<byte[],byte[]>getFamilybytefamily)所包含的修饰符与值的键值对byte[]byte[]qualifier)BytestoBytestablenameGetgGetBytetoBytesrow);ResultrowResult=table.get(g);rowResultgetValuefamilycolumn9)ResultScanner类ace户端获取值的接口。描述oidsesultnext)ResultScannerscannertable.getScanner(Bytes.toBytes(family));esultrowResultscannercolumn}河北工业大学——软件工程与理论实验室编辑：虾皮131)配置publicHBaseConfiguration()publicHBaseConfiguration(finalConfigurationc)ConfigurationconfnewConfiguration;thbasezookeeperquorumSlaveHadoopSlaveHadoopSlaveHadooprtyclientPortHBaseConfigurationcfgnewHBaseConfiguration(conf);2)创建表TableDescriptordescnsizepublicvoidaddFamily(finalHColumnDescriptorfamily)setInMemory，默认关闭；peHBaseAdminhAdminnewHBaseAdmin(hbaseConfig);DescriptortableName河北工业大学——软件工程与理论实验室编辑：虾皮14taddFamilynewHColumnDescriptor(“f1″));taddFamilynewHColumnDescriptor(“f2″));taddFamilynewHColumnDescriptor(“f3″));taddFamilynewHColumnDescriptor(“f4″));ateTablet3)删除表繁删除表。：HBaseAdminhAdminnewHBaseAdmin(hbaseConfig);if(hAdmin.tableExists(tableName)){eTabletableNameTabletableName}4)查询数据分为单条随机查询和批量查询。单条查询是通过rowkey在table中查询某一行的数据。HTable提供了get方法来完publicResultget(finalGetget)publicResultScannergetScanner(finalScanscan)publicGet(byte[]row)blicGetbyterowRowLockrowLockRowlock是为了保证读写的原子性，你可以传递一个已经存在Rowlock，否则HBase会自动生成一个新的rowlock。nn河北工业大学——软件工程与理论实验室编辑：虾皮15rrtRowsetStopRow：指定结束的行(不含此行)；setBatch：指定最多返回的Cell数目。用于防止一行中有过多的数据，导致publicResultnext()throwsIOException;publicResult[]next(intnbRows)throwsIOException;：newScanResultScannersstable.getScanner(s);sultrssStringrgetRowringkvgetColumn}}5)插入数据publicvoidput(finalPutput)throwsIOExceptionpublicvoidput(finalList<Put>puts)throwsIOException河北工业大学——软件工程与理论实验室编辑：虾皮16Put提供了3种构造方式：publicPut(byte[]row)licPutbyterowRowLockrowLockutputToCopy否写Log。默认是打开，关掉会提高性能，但是如果系统出现故障(负责插入的：eConfigtableNametablesetAutoFlushautoFlush;PutlpnewArrayListPutintcount000;bytebuffer=newbyte[1024];RandomrnewRandom();for(inti=1;i<=count;++i){PutpnewPutStringformatrow09d”,i).getBytes());sbufferp.add(“f1″.getBytes(),null,buffer);p.add(“f2″.getBytes(),null,buffer);p.add(“f3″.getBytes(),null,buffer);p.add(“f4″.getBytes(),null,buffer);addp河北工业大学——软件工程与理论实验室编辑：虾皮17ifi==0){tableputlp);clear}}6)删除数据publicvoiddelete(finalDeletedelete)publicDelete(byte[]row)publicDeletebyterowlongtimestampRowLockrowLockpublicDelete(finalDeleted)deleteFamily/deleteColumns：指定要删除的family或者column的数据。如果不调：baseConfigmytesttabledeleted);7)切分表publicvoidsplit(finalStringtableNameOrRegionName)elitregion：河北工业大学——软件工程与理论实验室编辑：虾皮18publicvoidsplit(StringtableName,intnumber,inttimeout)throwsException{ConfigurationconfnewConfigurationezookeeperquorumGlobalConfZOOKEEPERQUORUMtPortGlobalConfZOOKEEPERPORTHBaseConfigurationcfgnewHBaseConfiguration(conf);HBaseAdminhAdmin=newHBaseAdmin(cfg);toldsizetSystemcurrentTimeMilliswhiletrueablegetRegionsInfosizenfotheregionnumbersizeifsizenumberbreakfsizeoldsizeNamedsizesize}SystemcurrentTimeMillisttimeoutbreak;}Threadsleep*10);}}2.1开发环境Eclipseeclipse-jee-indigo-SR1-win32.zip|eclipse-jee-helios-SR2-win32.zipasehbasetargz河北工业大学——软件工程与理论实验室编辑：虾皮19配置Eclipse2.3测试样例s：ebuthbaseIOExceptionapachehadoopconfConfigurationapachehadoophbaseHBaseConfiguration河北工业大学——软件工程与理论实验室编辑：虾皮20apachehadoophbaseHColumnDescriptorgapachehadoophbaseHTableDescriptorgapachehadoophbaseclientHBaseAdminrgapachehadoophbaseutilBytespublicclassHelloHBase{publicstaticvoidmain(String[]args)throwsIOException{figurationconfHBaseConfigurationcreateasezookeeperquorumperty.clientPort","2181");eAdminadminnewHBaseAdminconfeDescriptorBytesametableDescriptorgetNamenDescriptorcolumnFamiliestableDescriptorsfor(HColumnDescriptord:columnFamilies){}}}12/03/2010:29:18INFOzookeeper.ZooKeeper:Client12/03/2010:29:18INFOzookeeper.ZooKeeper:ClientnvironmenthostnameZDLYVSTOFLZINFOzookeeper.ZooKeeper:Clientenvironment:java.version=1.6.0_31 10:29:18INFOzookeeper.ZooKeeper:Clientenvironment:piler=<NA>0:29:18INFOzookeeper.ZooKeeper:Clientenvironment:=Windows710:29:18INFOzookeeper.ZooKeeper:Clientenvironment:os.arch=x8612/03/2010:29:18INFOzookeeper.ZooKeeper:Clientenvironment:os.version=6.1河北工业大学——软件工程与理论实验室编辑：虾皮2110:29:18INFOzookeeper.ZooKeeper:Clientenvironment:=hadoop12/03/2010:29:18INFOzookeeper.ZooKeeper:ClientvironmentuserhomeCUsersAdministrator12/03/2010:29:18INFOzookeeper.ZooKeeper:ClientenvironmentuserdirEHadoopWorkPlatworkspaceHBaseDemo12/03/2010:29:18INFOzookeeper.ZooKeeper:Initiatingclientconnection,connectString.1.4:2181,:2181,192.168.5:2181sessionTimeout=180000tcherhconnection12/03/2010:29:18INFOzookeeper.ClientCnxn:Openingsocketconnectiontoserver.3:2181YVSTOFLZINFOzookeeper.ClientCnxn:SocketconnectionestablishedtoSlaveHadoop8.1.3:2181,initiatingsessionSlave1.Hadoop/:2181,sessionid=0x362a8d2a210004,negotiatedtimeout=aAPIoapiboyaoxueboyg：ebuthbaseilArrayListtilList河北工业大学——软件工程与理论实验室编辑：虾皮22apachehadoopconfConfigurationapachehadoophbaseHBaseConfigurationapachehadoophbaseHColumnDescriptorgapachehadoophbaseHTableDescriptorapachehadoophbaseKeyValuegapachehadoophbaseclientDeletergapachehadoophbaseclientGetgapachehadoophbaseclientHBaseAdmingapachehadoophbaseclientHTablergapachehadoophbaseclientPutgapachehadoophbaseclientResultrgapachehadoophbaseclientResultScannergapachehadoophbaseclientScanrgapachehadoophbaseutilBytespublicclassHBaseJavaAPI{taticConfigurationconfnullstatic{Configurationcreateconf.set("hbase.zookeeper.quorum",.168.1.5");confsethbasezookeeperpropertyclientPort}ticvoidcreateTableStringtableNameString[]columnFamilys)throwsException{HBaseAdminhAdminnewHBaseAdminconf;if(hAdmin.tableExists(tableName)){SystemoutprintlnitbleDescriptortableDescnewHTableDescriptortableNamefor(StringcolumnFamily:columnFamilys){tableDescaddFamilynewHColumnDescriptorcolumnFamily}abletableDesc河北工业大学——软件工程与理论实验室编辑：虾皮23Systemoutprintln}}publicstaticvoiddeleteTable(StringtableName)throwsException{HBaseAdminhAdminnewHBaseAdminconf;if(hAdmin.tableExists(tableName)){eTabletableNameabletableNameSystemoutprintlnSystemoutprintln);it}}blicstaticvoidaddRowStringtableNameStringrowtringcolumnFamilyStringcolumnStringvaluethrowsExceptionabletablenewHTableconftableNamenewPutBytestoBytesrowtestoBytescolumnFamilyBytestoBytescolumnvalueleputput}taticvoiddelRowStringtableNameStringrowthrowsException{abletablenewHTableconftableNameletedelnewDeleteBytestoBytesrowledeletedel}cstaticvoiddelMultiRowsStringtableNameStringrowsthrowsException{河北工业大学——软件工程与理论实验室编辑：虾皮24abletablenewHTableconftableNametnewArrayListDeletefor(Stringrow:rows){letedelnewDeleteBytestoBytesrow}ledeletelist}taticvoidgetRowStringtableNameStringrowthrowsException{abletablenewHTableconftableNameGetgetnewGetBytestoBytesrow));lttablegetgetfor(KeyValuerowKV:result.raw()){SystemoutprintnewString(rowKV.getRow())+"");tampSystemoutprintnewString(rowKV.getQualifier())+"");}}publicstaticvoidgetAllRows(StringtableName)throwsException{abletablenewHTableconftableNamescannewScanerresultstablegetScannerscanfor(Resultresult:results){for(KeyValuerowKV:result.raw()){SystemoutprintnewString(rowKV.getRow())+"");tampue}}}河北工业大学——软件工程与理论实验室编辑：虾皮25publicstaticvoidmain(String[]args){try{gtableNamestudentStringcolumnFamilysinfocourse"};avaAPIcreateTabletableNamecolumnFamilysHBaseJavaAPIaddRowtableNamexiapiinfo"age","20");HBaseJavaAPIaddRowtableNamexiapiinfo"sex","boy");HBaseJavaAPIaddRowtableNamexiapicourse"china","97");HBaseJavaAPIaddRowtableNamexiapicourse"math","128");HBaseJavaAPIaddRowtableNamexiapicourse"english","85");HBaseJavaAPIaddRowtableNamexiaoxueinfo"age","19");HBaseJavaAPIaddRowtableNamexiaoxueinfo"sex","boy");HBaseJavaAPIaddRowtableNamexiaoxuecourse"china","90");HBaseJavaAPIaddRowtableNamexiaoxuecourse"math","120");HBaseJavaAPIaddRowtableNameqingqinginfo"age","18");HBaseJavaAPIaddRowtableNameqingqinginfo"sex","girl");ursemathSystemoutprintln;JavaAPIgetRowtableNamexiapiSystemoutprintln;APIgetAllRowstableNameSystemoutprintln;JavaAPIdelRowtableNamexiapiAPIgetAllRowstableNameSystemoutprintln;String[]rows={"xiaoxue","qingqing"};vaAPIdelMultiRowstableNamerowsAPIgetAllRowstableName河北工业大学——软件工程与理论实验室编辑：虾皮26SystemoutprintlnAPIdeleteTabletableNamecatchExceptionerrStackTrace}}}27INFOzookeeper.ZooKeeper:Clientenvironment:03/2013:42:27INFOzookeeper.ZooKeeper:Clientenvironment:hostnameZD1LYV4S3T7OFLZ27INFOzookeeper.ZooKeeper:Clientenvironment:java.version=1.6.0_317INFOzookeeper.ZooKeeper:Client 2:28INFOzookeeper.ClientCnxn:EventThreadshutdowne河北工业大学——软件工程与理论实验室编辑：虾皮27hetionconnectString81,:2181,:2181out9INFOzookeeper.ClientCnxn:EventThreadshutdown29INFOclient.HBaseAdmin:Starteddisableofstudent30INFOclient.HBaseAdmin:Disabledstudent2:31INFOclient.HBaseAdmin:DeletedstudentRunASJavaApplication。DFSMapReduce调用HBase进行特殊的处理，这样能够充分结合HBase分布式大型数据库和MapReduce并行计算的优点。每个阶段提供了相应的类用来处理表数据。对。TableInputFormat类将数据表按照Region分割成split，既有多少个Regions就有多个河北工业大学——软件工程与理论实验室编辑：虾皮283)OutputFormat类：HBase实现的TableOutputFormat将输出的<key,value>对写到java.lang.ClassNotFoundException:apachehadoophbasemapreduceTableOutputFormatenvironmentportHBASEHOMEusrhbaseTHHBASEHOMEhbasejar$HBASE_HOME/hbase-0.92.0-tests.jar:$HBASE_HOME/conf:zookeeperjar例(1)河北工业大学——软件工程与理论实验室编辑：虾皮29首先从文件中收集数据，在shuffle完成之后进行统计并计算，最后将计算结构存：ebuthbaseIOExceptionilIteratortilStringTokenizerapachehadoopconfConfigurationapachehadoopfsPathapachehadoophbaseHBaseConfigurationapachehadoophbaseHColumnDescriptorgapachehadoophbaseHTableDescriptorgapachehadoophbaseclientHBaseAdminrgapachehadoophbaseclientPutrgapachehadoophbasemapreduceTableOutputFormatrgapachehadoophbasemapreduceTableReducerrgapachehadoophbaseutilBytesapachehadoopioIntWritablegapachehadoopioLongWritableapachehadoopioTextgapachehadoopioNullWritableapachehadoopmapreduceJobgapachehadoopmapreduceMappertorgapachehadoopmapreducelibinputFileInputFormattorgapachehadoopmapreducelibinputTextInputFormatpublicclassWordCountHBase{cclassMapextendsMapperLongWritableTextTextIntWritable{vatefinalstaticIntWritableonenewIntWritableteTextwordnewTextcvoidmapLongWritablekeyTextvalueContextcontextowsIOExceptionInterruptedExceptionStringTokenizeritr=newStringTokenizer(value.toString());while(itr.hasMoreTokens()){wordsetitrnextToken);wordone河北工业大学——软件工程与理论实验室编辑：虾皮30}}}classReduceextendsTableReducer<Text,IntWritable,NullWritable>{voidreduceTextkeyIterableIntWritablevaluesmIntWritableiteratorvaluesiteratorwhile(iterator.hasNext()){tornextget}putnewPutBytestoByteskeytoStringddBytestoBytescontentBytestoBytescountesStringvalueOfsumiteNullWritablegetput}}icvoidcreateHBaseTableStringtableNamethrowsIOException{leDescriptorhtdnewHTableDescriptortableNameumnDescriptorcolnewHColumnDescriptorcontentolfigurationconfHBaseConfigurationcreateconf.set("hbase.zookeeper.quorum",.168.1.5");confsethbasezookeeperproperty.clientPort","2181");河北工业大学——软件工程与理论实验室编辑：虾皮31seAdminhAdminnewHBaseAdminconfif(hAdmin.tableExists(tableName)){eTabletableNameabletableName}eablehtd}publicstaticvoidmain(String[]args)throwsException{tableNamewordcountrdCountHBasecreateHBaseTabletableNameonfigurationconfnewConfigurationconfsetmapredjobtracker.2:9001");conf.set("hbase.zookeeper.quorum",.168.1.5");nfsethbasezookeeperpropertyclientPortbleOutputFormatOUTPUTTABLEtableNameJobjobnewJobconf,"NewWordCount");yClassWordCountHBaseclasserClassMapclasscerClassReduceclasstputKeyClassTextclasstputValueClassIntWritableclassFormatClassTextInputFormatclassutFormatClassTableOutputFormatclassmataddInputPathjobnewPathhbaseinput河北工业大学——软件工程与理论实验室编辑：虾皮32索引表索引表SystemexitjobwaitForCompletiontrue);}}doop例(2)HBase索引主要用于提高HBase中表数据的访问速度，有效地避免了全表扫描(多数效。相反，如果经常被查询的列作为行健、行健作为列重新构造一张表，即可实现根据列关数据所在的行，这就是索引。显然索引表仅需要包含一列，所以索引表原表河北工业大学——软件工程与理论实验室编辑：虾皮33Table.ROOT.TableTable.ROOT.Table健er1peterpeter@bilities2irohiro@bendtimeandspace3eroescomhnowhowthingswork4ereheroescomeal5oahnoah@aththepeoplewithablities.META.河北工业大学——软件工程与理论实验室编辑：虾皮34：ebuthbaseByteArrayOutputStreamoDataOutputStreamIOExceptionilHashMapapachehadoopconfConfigurationapachehadoophbaseHBaseConfigurationapachehadoophbaseHColumnDescriptorgapachehadoophbaseHTableDescriptorgapachehadoophbaseclientHBaseAdmingapachehadoophbaseclientHTablergapachehadoophbaseclientPutgapachehadoophbaseclientResultgapachehadoophbaseclientScangapachehadoophbaseioImmutableBytesWritablergapachehadoophbasemapreduceMultiTableOutputFormatrgapachehadoophbasemapreduceTableInputFormatorgapachehadoophbaseutilBasergapachehadoophbaseutilBytespachehadoopioWritableapachehadoopmapreduceJobgapachehadoopmapreduceMapperpublicclassIndexBuilder{rivatestaticfinalbyteINDEXCOLUMNBytestoBytesINDEXrivatestaticfinalbyteINDEXQUALIFIERBytestoBytesROWcclassMapextendsMapper<ImmutableBytesWritable,Result,ImmutableBytesWritable,Writable>{；后者用户作为索引表的表名HashMapbyteImmutableBytesWritableindexes河北工业大学——软件工程与理论实验室编辑：虾皮35efamilydmapImmutableBytesWritablekeyResultvalueualifierindexgetKeyytesWritabletableNameindexgetValueenewValuevaluegetValuefamilyqualifierif(newValue!=null){DEXROWwPutnewValuedINDEXCOLUMNINDEXQUALIFIERkeygettableNameput}}}setupMapper只在任务初始化时执行一次dvoidsetupContextcontextthrowsIOExceptionInterruptedExceptiononfigurationconfcontextgetConfigurationingtableNameconfgetindextablenametringfieldsconfgetStringsindexfieldsringfamilyNameconfgetindexfamilynamefamily=Bytes.toBytes(familyName);snewHashMapbyteImmutableBytesWritablefor(Stringfield:fields){河北工业大学——软件工程与理论实验室编辑：虾皮36estoBytesfieldytesWritable}}}//初始化示例数据表——“heroes”throwsIOException{leDescriptorhtdnewHTableDescriptortableNamelumnDescriptorcolnewHColumnDescriptorinfoolseAdminhAdminnewHBaseAdminconfif(hAdmin.tableExists(tableName)){eTabletableNameabletableName}eablehtdabletablenewHTableconftableNameSystemoutprintln;addRowtableinfo"name","peter");addRowtableinfo"email","peter@");addRowtableinfo"power","absorbabilities");addRowtableinfo"name","hiro");addRowtableinfo"email","hiro@");addRowtableinfo"power","bendtimeandspace");addRowtableinfo"name","sylar");addRowtableinfo"e