Hadoop开发者入门专刊

bbshadooporcom-hadoopbbshadooporcom-hadoop技术论坛2010入门专刊1-见)贺湘辉(小米)王磊(beyi)柏传杰(飞鸿雪泥)Spork贡献力量的开源者们的共同努力，才有了Hadoop自诞生时的倍受关注到现在p能聚到一起，为了一个共同的爱好策划这本杂志，这本身就是Hadoop魅力的体现。当然，也要感谢大师DougCutting和Hadoop社区的开源者们，因为有了您，这里才变得如此精彩！《Hadoop开发者》编辑组2010-1-271Hadoop介绍2Hadoop在国内应用情况Hadoop源代码eclipse编译教程Windows上安装Hadoop教程LinuxHadoop教程eHadoopindowsCygwinHBaseNutch与Hadoop的整合与部署Windowseclipse调试Hive教程38Hive应用介绍42Hive执行计划解析ShuffleSort53海量数据存储和计算平台的调试器研究探讨MapReduce模型的改进eclipseHadoop框架表关联在MapReduce上的实现Hadoop计算平台和Hadoop数据仓库的区别bbshadooporcom-hadoop技术论坛Hadoop介绍HDFSGoogleGFS分布式文件系统，它能够提供高吞InfoWordMapReduce的冠军。MapReduce是大规模数据 TBMapReduce是它的主要思想，来源于函数式编程语言，它的原TB的计算，常见的应用包括：HadoopMapReduce的实现也采用r1-bbshadooporcom-hadoop技术论坛Hadoop在国内应用情况司httpbbshadooporcomthread8-1-1.html分布op前三讯，杭州有使用力量。使用城市。2-bbshadooporcom-hadoop技术论坛Hadoop源代码eclipse编译教程/repos/asf/hadoop/common/trunk，httpsvnapacheorgreposasfhadoopcommon录下包括了很多非源代准备编译环境Hadoop代码版本2.联网2.3.java2.4.Ant和Cygwindp2.5.Eclipse3-bbshadooporcom-hadoop技术论坛t4-bbshadooporcom-hadoop技术论坛，操作结果如下图所示：dHadoop各成员都需要编译成jar，所以做如下图所示的一个修改：5-bbshadooporcom-hadoop技术论坛请注意：如果上图所示的菜单中的BuildAutomatically被勾中，则在common的右键6-.联网t3.编译生成文件位置ndowsHadoop1.安装JDKCNdownloadmanualjsp2.安装Cygwin要将Cygwin安装目录及子目录的拥有者(Owner)。Cygwin安装程序下载地址：/setup.exe，当然也可以从/setup.exe下载，本教程下载的是Cygwin1.7.1。setup.exe的存 bbshadooporcom-hadoop技术论坛8-bbshadooporcom-hadoop技术论坛d9-bbshadooporcom-hadoop技术论坛3.配置环境变量4.安装sshd服务10-bbshadooporcom-hadoop技术论坛5.启动sshd服务6.配置ssh登录cd~/..ssh/cpid_rsa.pubauthorized_keys重新运行Cygwin，执行sshlocalhost命令，在第一次执行时，会有如下图bbshadooporcom-hadoop技术论坛这个错误暂无解决办法，问题的解决情况，可关注Hadoop技术论坛中的贴：7.下载hadoop安装包/apache-mirror/hadoop/core/hadoop-0.20.1/hadoop-0.1.tar.gze8.安装hadoopn目录)下，如下图所示：修改hadoop-env.shexportJAVA_HOME=/cygdrive/d/java/jdk1.6.0_13修改core-site.xml bbshadooporcom-hadoop技术论坛修改hdfs-site.xmlsitexml它修改。修改mapred-site.xml9.启动hadoop作者：小米Email：gshexianghui@126.come13-bbshadooporcom-hadoop技术论坛Ubuntu04+jdk1.6+hadoop-0.20.1(三台实体机)HexianghuiNameNode、master、jobTrackerexianghuiDataNode、slave、taskTrackerhexianghui5DataNode、slave、taskTrackercker台做DataNode,slave,taskTracker。关于这几个概念，可以参考Hadoop的官方文档更新源修改(方便以后更新提高速度，教育网可以参考如下网址)：/itdreams2009/blog/item/dae1cf1208b53e8a6438dbac.html用户hexianghui。/blog/static/680550312008481822419/安装openssh-server：$sudoapt-getinstallopenssh-server窗口：14-bbshadooporcom-hadoop技术论坛正常进入，就可以实现远程登录和管理了。$ssh-keygen-tdsa-P''-f~/.ssh/id_dsa，文件)：$cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys。完成后可以实现无密码登录本机：$sshlocalhost。$scpid_dsa.pubhexianghui@:/home/hexianghui/b.登录，执行$catid_dsa.pub>>.ssh/authorized_keysauthorized_keys：$chmod600authorized_keys。防火墙$sudoufwdisable安装。本例的安装路径为/home/hexianghui/jdk1.6.0_14。exportJAVA_HOME=/home/hexianghui/jdk1.6.0_14exportJRE_HOME=/home/hexianghui/jdk1.6.0_14/jreexportCLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATHexportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH/apache-mirror/hadoop/core/hadoop-0.20.1/hadoop-0.20.1.tar.gz解压：$tar–zvxfhadoop-0.20.1.tar.gzortHADOOPHOMEhomehexianghuihadoop15-bbshadooporcom-hadoop技术论坛THHADOOPHOMEbinPATH$gedithadoop-env.sh$exportJAVA_HOME=/home/hexianghui/jdk1.6.0_14masters:.3texmlresitexml<configuration><!globalproperties--><property><name>hadoop.tmp.dir</name><value>/home/hexianghui/tmp</value><description>Abaseforothertemporarydirectories.</description></property><!--filesystemproperties--><property><name></name><value>hdfs://:9000</value></property></configuration><configuration><property><name>dfs.replication</name><value>1</value></property></configuration>mapred-site.xml:<configuration><property><name>mapred.job.tracker</name><value>:9001</value></property></configuration>16-hadoop-0.20.1/bin，首先格式化文件系统：$hadoopnamenode–formattallsh查看集群状态：$hadoopdfsadmin-report$echo“HelloWorldByeWorld”>file01$echo“HelloHadoopGoodbyeHadoop”>file02$hadoopfs–copyFromLocal/home/hexianghui/soft/file0*input$hadoopjarhadoop-0.20.1-examples.jarwordcountinputoutput$hadoopfs-catoutput/part-r-00000ByeBye17-bbshadooporcom-hadoop技术论坛Goodbye1Hadoop2Hello2World2应的解决方法：File/home/hexianghui/tmp/mapred/system/couldonlybe闭防火墙。：把safemode置于off状态：$hadoopdfsadmin–safemodeleaveedownnetsofthtm问题以及解决方法发送到gshexianghui@126.com或者贴在hadoopor技术论坛上httpbbshadooporcom18-bbshadooporcom-hadoop技术论坛在Windows上使用eclipse编写Hadoop应用程序1.前言2.创建Java工程smain3.配置编译参数9bbshadooporcom-hadoop技术论坛core.jar”添加进来。4.源代码publicclassDFSOperator{/***/publicstaticvoidmain(String[]args){//TODOAuto-generatedmethodstubConfigurationconf=newConfiguration();try{FileSystemfs=FileSystem.get(conf);peratortxttrue);Pathf=newPath("hdfsperatortxttrue);FSDataOutputStreamos=fs.create(f,inti=0;forii0000;++i)os.writeChars("test");os.close();}catch(IOExceptione){//TODOAuto-generatedcatchblocke.printStackTrace();}}}5.编译生成JAR0bbshadooporcom-hadoop技术论坛1bbshadooporcom-hadoop技术论坛按上图所示，设置好JAR包的文件名和存放位置，然后点击“Finish”按钮，生成ratorjar6.测试运行bin2bbshadooporcom-hadoop技术论坛dfsoperator.jar是否存在：ls(2)查看Hadoop根目录下是否存在dfs_operator.txt文件：./hadoopfs-ls/xt./hadoopjar./dfs_operator.jarDFSOperator(4)再查看Hadoop根目录下是否存在dfs_operator.txt文件：./hadoopfs-ls/dfsoperatortxthadoopfs-cat/dfs_operator.txt7.结束3bbshadooporcom-hadoop技术论坛作者：飞鸿雪泥Email：jaguar13@1.简介httplabsgooglecompapersbigtable-osdi06.pdf)。2.目的3.安装与配置JDK1.6(或以上版本)Cygwin2.5Hadoop0.20.xHBase0.20.x3.1.Java、Cygwin、SSH3.2.HadoopexportJAVA_HOME=/cygdrive/c/Progra~1/Java/jdk1.6.0_01Eln-s/cygdrive/c/Program\Files/Java/jdk1.6.0_01\/usr/local/jdk1.6.0_01exportJAVA_HOME=/usr/local/jdk1.6.0_014bbshadooporcom-hadoop技术论坛opbin/hadoopdfs–mkdirhbasetmp3.3.HBaseexportJAVA_HOME=/cygdrive/c/Progra~1/Java/jdk1.6.0_01exportHBASE_IDENT_STRING=localhost2.将$HBASE_HOME/conf/hbase-default.xml中的所有内容，拷贝到<property><name>hbase.rootdir</name><value>hdfs://localhost:8888/hbase</value><description>Thedirectorysharedbyregionservers.Shouldbefully-qualifiedtoincludethefilesystemtouse.E.g:hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description></property><property><name>hbase.tmp.dir</name><value>hdfs://localhost:8888/tmp</value><description>Temporarydirectoryonthelocalfilesystem.</description></property><property><name>hbase.zookeeper.quorum</name><value></value><description>CommaseparatedlistofserversintheZooKeeperQuorum.Forexample,",,".Bydefaultthisissettolocalhostforlocalandpseudo-distributedmodesofoperation.Forafully-distributedsetup,thisshouldbesettoafulllistofZooKeeperquorumservers.IfHBASE_MANAGES_ZKissetinhbase-env.shthisisthelistofserverswhichwewillstart/stopZooKeeperon.</description>5bbshadooporcom-hadoop技术论坛</property>4.测试确安装。put'test','row1','data:1','value1'put'test','row2','data:2','value2'put'test','row3','data:3','value3'65.总结7bbshadooporcom-hadoop技术论坛作者：若冰Email：pi.bingfeng@初始环境准备：master：,hostname:crawler径为:/usr/java/jdk1.6.0_11scproot@:/root/.ssh/id_rsa.pub/root/.ssh/1_rsa.pubcat1_rsa.pub>>authorized_keys.vi/etc/hosts:lercrawler实验步骤:datanutchconfhadoopsitexml，简单的配置信息如下：<?<?xmlversion="1.0"?>8bbshadooporcom-hadoop技术论坛<?xml-stylesheettype="text/xsl"href="configuration.xsl"?><configuration><property><name></name><value>hdfs://:54310</value></property><property><name>mapred.job.tracker</name><value>hdfs://:54311</value><description>jobtraker的配置，master的IP地址加端口</description></property><property><name>dfs.replication</name><value>1</value><description>replicationofdfssystem，测试阶段可以设为1</description></property><property><name>hadoop.tmp.dir</name><value>/data/hadoopdata/tmp</value><description>Abaseforothertemporarydirectories.</description></property><property><name>dfs.data.dir</name><value>/data/hadoopdata/data</value><description>DetermineswhereonthelocalfilesystemanDFSdatanodeshouldstoreitsblocks.Ifthisisacomma-delimitedlistofdirectories,thendatawillbestoredinallnameddirectories,typicallyondifferentdevices.Directoriesthatdonotexistareignored.</description></property><property><name>.dir</name><value>/data/hadoopdata/name</value><description>DetermineswhereonthelocalfilesystemtheDFSnamenodeshouldstorethenametable(fsimage).Ifthisisacomma-delimitedlist9bbshadooporcom-hadoop技术论坛ofdirectoriesthenthenametableisreplicatedinallofthedirectories,forredundancy.</description></property></configuration>pk#Thejavaimplementationtouse.Required.exportJAVA_HOME=/usr/java/jdk1.6.0_11esse修改/data/nutch-1.0/conf/crawl-urlfilter.txt文件如下：##accepthostsinMY.DOMAIN.NAME#+^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/+^http://([a-z0-9]*\.)*/<property><name></name><value>nutch-1.0</value><description></description></</property>bin/hadoopfs–mkdircrawlbin/hadoopfs–mkdirrooturlalbin/hadoopfs–putroot.txtrooturl0bbshadooporcom-hadoop技术论坛jar-xvfnutch-1.0.tar/usr/local/tomcat/webapps/nutchhdir<<property><name>searcher.dir</name><value>/data/search</value><description></description><</property>经验总结与注意事项：编后语1.前言否则请先掌握安装和编译方法，才继续往下浏览。2.参考资料3.安装Hadoop和编译Hive1bbshadooporcom-hadoop技术论坛se4.安装Hive目录下(可不复制SVN目录；Ehadoopsrchivebuildjar到E:\hadoop\run\hive\lib目LICENSEjareclipsehiveHive已经编译通过)，点击菜单“Run”,2bbshadooporcom-hadoop技术论坛tion3bbshadooporcom-hadoop技术论坛Apply钮，如下图所示：4bbshadooporcom-hadoop技术论坛在上图所示对话框中，选择“AddExternalFolder”，将“E:\hadoop\run\hive\conf”添5Scannersc=newScanner(System.in);whi1e(true){6bbshadooporcom-hadoop技术论坛System.out.print(curPrompt+">");line=sc.nextLine();if(null==line)break;F7：从当前函数内跳到上一级调用F8：一直执行到断点CtrlShiftB除断点html5.编后语要供从C/C++等非java语言转过来的Hadoop开发者。欢迎大家加入Hadoop技术论坛()多多交流问题和分享经验。7bbshadooporcom-hadoop技术论坛Hive应用介绍作者：李均Email：fiberlijun@1.Hive简介eHive2.Hive安装和配置hadoop-0.19.1集成了hive，安装配置起来比往常版本要更加傻瓜化。exportHADOOP=/usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.19.1ExportIVHOME=/usr/local/distribute-hadoop-boss/hadoopenv/hadoop-0.19.1/contrib/hiveOMEbinhivemysql的shell，hive-f参数直接执行某个文件里的查询语句。8bbshadooporcom-hadoop技术论坛3.使用举例3.1.支持的字段数据类型Integerssmallisbytesmediumisbytesandbigis8bytes)2)Floatingpointnumbers(SingleandDoubleprecision)Strings4)Datetime5)Boolean3.2.创建Table，load数据CREATETABLEtcss(domain_idINT,log_timeSTRING,log_dateSTRING,log_typeINT,uinBIGINT)ROWFORMATDELIMITEDFIELDSTERMINATEDBY','LOCATION'/user/hivetest/tcss';缺省配置下，Hive会在hadoop文件系统创建目录/user/hive/warehouse/tcss来存放数据hadoopfs-put./tcss_20090301_qq.small/user/hivetest/tcss/small1hadoopfs-put./tcss_20090301_qq.small/user/hivetest/tcss/small200:34:44,20090301,1001,1291162103-0100:34:46,20090301,0,8418428099-03-0100:34:48,20090301,10180,22395900100:34:50,20090301,10180,867110912INSERTOVERWRITETABLEtcssoutselecttcss.*fromtcss;INSERTOVERWRITEDIRECTORY'/tmp/reg_4'FIELDDELIMITER','selecttcss.*fromtcss;3.3.表查询selectcount(tcss.uin)fromtcss;9bbshadooporcom-hadoop技术论坛useractiveSELECTuserWHEREuseractivetrueInnerJOIN查询FROMpage_viewpvJOINuseruON(pv.userid=u.id)INSERTOVERWRITETABLEpv_usersSELECTpv.*,u.gender,u.ageWHEREpv.date=2008-03-03;INFROMpage_viewpvJOINuseruON(pv.userid=u.id)JOINfriend_listfON(u.id=f.uid)INSERTOVERWRITETABLEpv_friendsSELECTpv.*,u.gender,u.age,ff.riendsWHEREpv.date=2008-03-03;3.4.聚合查询FROMpv_usersINSERTOVERWRITETABLEpv_gender_sumSELECTpv_users.gender,count(DISTINCTpv_users.userid)GROUPBYpv_users.gender;FROMpv_usersINSERTOVERWRITETABLEpv_gender_aggSELECTpv_users.gender,count(DISTINCTpv_users.userid),count(),sum(DISTINCTpv_users.userid)GROUPBYpv_users.gender;3.5.其他高级查询UnionallmmapreducescriptsupsCasestatementHive.0支持)4.Partition介绍0bbshadooporcom-hadoop技术论坛4.1.准备每个目录有1个数据文件。具体见下图每个目录有1个数据文件,具体见下图。接下来我们要4.2.如何PartitionCREATETABLEqzone_partion_t(logidstring,datetime1string,uin1bigint,datetime2string,uin2bigint,ipstring,logtypeint,mdatestring)PARTITIONEDBY(partdateDATETIME)ROWFORMATDELIMITEDFIELDSTERMINATEDBY'\t'LOCATION'/user/distribute-hadoop-boss/ISD_QZONE';)把hadoop某个目录的文件按Partition插入表先建一个临时表映射到目录/ISD/20090717，表结构和上面的一致，只是没有partitionCREATEEXTERNALTABLEqzone_20090717(logidstring,datetime1string,uin1bigint,datetime2string,uin2bigint,ipstring,logtypeint,mdatestring)ROWFORMATDELIMITEDFIELDSTERMINATEDBY'\t'LOCATION'/user/distribute-hadoop-boss/ISD/20090717';FROMqzone_20090717tINSERTOVERWRITETABLEqzone_partion_tPARTITION(partdate='2009-07-17')select*;过程其实是一个拷贝过程通过map完成，见下图。如果目录内容很大会有一通过上述操作我们把./ISD/20090717到./ISD/20090722六个目录的数据按日期rtition1bbshadooporcom-hadoop技术论坛4.3.基于Partition的查询1.针对某一个partition日期查询，hive只会对partition对应的这个目录做selectcount(1)fromqzone_partion_twhereqzone_partion_t.partdate='2009-07-17';selectcount(1)fromqzone_partion_twhereqzone_partion_t.partdate>='2009-07-01'andpartdate<='2009-07-30';Hive执行计划解析1.前言2.流程分析2bbshadooporcom-hadoop技术论坛解析生成执行计划INSERTOVERWRITETABLEdest_j1SELECTsrc1.key,src2.valueFROMsrcsrc1JOINsrcsrc2ON(src1.key=src2.key);3BaseSemanticAnalyzerBaseSemanticAnalyzerParseDriverContextTokenStreamHiveLexerHiveParserXTokenRewriteStreamHiveLexerX+setTreeAdaptor(in:CommonTreeAdaptor)CommonTreeAdaptorDriverSemanticAnalyzerFactoryecute+get():BaseSemanticAnalyzerASTNode经历如下几个处理：SQL抽象语法树->QB(QueryBlock)->逻辑计划树(Operator)->执行计划树HiveParserstatement_return-HiveParserANTLRNoCaseStringStreamn4txdoPhase1((ASTNode)insertClauseselectClausefromClause(whereClause)?(groupByClause)?(orderByClause)?(clusterByClause)?(distributeByClause)?(sortByClause)?(limitClause)?->^(TOK_QUERYfromClause^(TOK_INSERTinsertClauseselectClause(whereClause)?(groupByClause)?(orderByClause)?(clusterByClause)?(distributeByClause)?sortByClause)?(limitClause)?))|selectStatement)2.抽象语法树->QB->逻辑计划树->执行计划树(Task)1)逻辑计划生成publicvoiddoPhase1(ASTNodeast,QBqb,Phase1Ctxctx_1){QBParseInfoqbp=qb.getParseInfo();booleanskipRecursion=false;if(ast.getToken()!=null){skipRecursion=true;etTypetorderpdefault:skipRecursion=false;break;}}if(!skipRecursion){//迭代子查询intchild_count=ast.getChildCount();++child_pos){for(intchild_pos=0;child_pos<child_count;++child_pos){//递归tgetChildchildpos}}}5息息否是}}2)获取元数据据库中作为元数据。EXPLAININSERTOVERWRITETABLEdest_j1SELECTsrc1.key,src2.valueFROMsrcsrc1JOINsrcsrc2ON(src1.key=src2.key);能得到如下结果：SStage1isarootstageStage-0dependsonstages:Stage-1Stage:Stage-1apReduceAlias>MapOperatorTree:srcableScanaliassrc2educeOutputOperatorkeyexpressionsexprkeytypeintsortorder+Mapreducepartitioncolumns:6bbshadooporcom-hadoop技术论坛exprkeytypeinttag1valueexpressionsexprvaluetypestringsrcableScanaliassrc1educeOutputOperatorkeyexpressionsexprkeytypeintsortorder+Mapreducepartitioncolumns:exprkeytypeinttag0valueexpressionsexprkeytypeintReduceOperatorTreeoinOperatorconditionmapInnerJoin0to1conditionexpressions0{VALUE._col0}1{VALUE._col1}outputColumnNames:_col0,_col3electOperatorexpressionsexpr:_col0typeintexpr:_col3typestringoutputColumnNames:_col0,_col1ileOutputOperatorcompressedfalseGlobalTableId1tableinputformatorgapache.hadoop.mapreduce.lib.input.TextInputFormatoutputformatorgapachehadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat7bbshadooporcom-hadoop技术论坛serdeorgapachehadoophive.serde2.lazy.LazySimpleSerDename:dest_j1StageStage-0oveOperatortablesreplacetruetableorgapachehadoopmapreducelibinputTextInputFormatrmatorgapachehadoophiveqlioHiveIgnoreKeyTextOutputFormathadoophiveserdelazyLazySimpleSerDenamedestj1Stage-1isarootstageStage-0dependsonstages:Stage-1SELECTsrc1.key,src2.valueFROMsrcsrc1JOINsrcsrc2ON(src1.key=src2.key);insertoverwriteintodest_j1；r jttable:inputformat:org.apache.hadoop.mapreduce.lib.input.TextInputFormatoutputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormatserde:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDename:dest_j18bbshadooporcom-hadoop技术论坛outputformat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat可t一步验证:ProtectedstaticclassIgnoreKeyWriterextendsRecordWriter{publicsynchronizedvoidwrite(Kkey,Vvalue)throwsIOException{this.mWriter.write(null,value);}}MapReduce用MoveTask的处理就行，在处理之前会去检查输入文件是否符db.loadTable(newPath(tbd.getSourceDir()),tbd.getTable().getTableName(),tbd.getReplace(),newPath(tbd.getTmpDir()));if(work.getOutputs()!=null)work.getOutputs().add(newWriteEntity(table));3.编后语9bbshadooporcom-hadoop技术论坛rk0bbshadooporcom-hadoop技术论坛spillpercentiosortspillpercent在把缓冲区的数据写到磁盘前，会对它进行一个二次快速排序，首先根据数据所属的er磁盘前运行以压缩数据。1bbshadooporcom-hadoop技术论坛对写入到磁盘的数据进行压缩(这种压缩同Combiner的压缩不一样)通常是一个很好的方法，因为这样做使得数据写入磁盘的速度更快，节省磁盘空间，并减少需要传送到press.map.output为true启用该功能。压缩所使用的库由格式名分卷EFLATE无EFLATEEFLATEEFLATEs个数由tasktracker.http.threads控制，这个设定是针对每一个TaskTracker的，并不是Map可以增大以提升数据传输速率。现在让我们转到Shuffle的Reduce部分。Map的输出文件放置在运行Map任务的mapred.reduce.parallel.copies来改变线程数，默认是5。psReducer可能会失败。它们会在整个作业完成后，JobTracker除的时候才去删除。由mapred.job.shuffle.input.buffer.percent控制，制定了用于此目的的堆内存的百分 (由mapred.job.shuffle.merge.threshold控制)，或者达到了Map输出的阀值大小(由ergethresholdspill成。假设这里有50个Map输出(可能有保存在内存中的)，并且归并因子是10(由2bbshadooporcom-hadoop技术论坛的e接写到输出文件系统，一般是HDFS。在HDFS中，因为TaskTracker节点也运行着一个de海量数据存储和计算平台的调试器研究1.前言2.集群结构两类节点：调度节点和工作节点。3bbshadooporcom-hadoop技术论坛3.适用场景下应用场景：需要充分利用CPU等硬件资源需要多Job并行运行需要设置不同的Job运行级别对紧急情况(如领导要求出某份数据)的支持4.Job和Task5.Job优先级b5.1.建议优先级。5.2.实际优先级5.3.绝对优先级资源。优先级类型性值6.时间片和Task操作系统的调度器是有时间片概念的，以达到并多进程并发运行的目的。同样，MapReduce的调度也可以借用这个思想，问题的关键演变为如何确定时间片。Task作为一个调度时间片，是本调度器最核心的思想。7.调度准则4bbshadooporcom-hadoop技术论坛E8.性能估计9.内存估计。。s5bbshadooporcom-hadoop技术论坛MapReduce1.需要什么？问题：2.Map-Reduce模型写MapReduce代码的程序员来保证，这也使得MapReduce的调度器无法实现得相当灵活，多调度策略无法使用，而只能分集群多调度器3.Map-Balance-Reduce模型6bbshadooporcom-hadoop技术论坛的作用就是保证reduce的输入是均衡的，同时保证语义不变，因此用户需要实publicclassMapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>{protectedvoidmap(KEYINkey,VALUEINvalue,Contextcontext);}publicclassReducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>{protectedvoidreduce(KEYINkey,Iterable<VALUEIN>values,Contextcontext);}publicclassBalancer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>{protectedvoidbalance(KEYINkey,VALUEINvalue,Contextcontext);}7bbshadooporcom-hadoop技术论坛e4.MBRvsMR4.1.MBR优点一文扩展能力4.2.MBR缺点balance，存在性能降低的可能运行eclipse编译出的Hadoop框架rhadoophdfsSNAPSHOTjar8bbshadooporcom-hadoop技术论坛rwebappstoCLASSPATHif-d"$HADOOP_HOME/webapps"];thenfiforfin$HADOOP_HOME/hadoop-*-core.jar;doHforfinHADOOP_HOME/lib/*.jar;doupgrade命令，然后再重启Hadoop，一切就OK了。MapReduce若冰问题描述9bbshado