子计画自动化大量资料切割与整合

子計畫：自動化大量資料切割與整合國立高雄大學電機工程學系賴智錦/詹奇峰第三季成果Whathadbeenexpectedtodo以修正後的資料切割與重組演算法，設計多引線(multi-thread)的版本。以C或類似語言進行單機系統模擬。修正資料切割與重組有關之演算法。Whatwereachieved

瞭解分散式工具Hadoop中MapReduce的執行原理。以Java語言進行單機系統模擬。針對單一版本的Hadoop，進行多引線的演算法設計。Anymismatches

尚未進行更多樣的多引線分析。尚未進行其他Hadoop版本的多引線API設計。第三季成果Anydifficulties多引線之API開發，發展還不算完全。由於Hadoop版本的不同，將會造成API的限制設計。Futuretasks若需投入更多樣的分析，需從自行設計多引線API下手。對於multi-thread的設計原理與應用，需更深入瞭解。針對不同Hadoop版本的多引線之API進行分析。Comments

需分析範例的特性，是否適宜於雲端運算的multi-thread設計。第三季成果MapperRecord*MultithreadedMapperMultithreadedMapper.RunnerMultithreadedMapper.SubMapRecord*Mapper.ContextThread1Thread2Thread3Thread4…ThreadxHadoopMultithreadFramework第三季成果Import…//LoadHadoopAPIClassMapRed{

ClassMapper…{

//MapCode

}

ClassReducer…{

//ReduceCode } main(){

JobConfJob=newJobConf(“MapRed.class”); Job.setMapperClass(Mapper.class); Job.setReduceClass(Reducer.class);

FileInputFormat.setInputPaths(Job,newPath(args[0]));

FileOutputFormat.setOutputPath(Job,newPath(args[1])); JobClient.runJob(Job);//OtherCodeSet}}Reduce…Main…Map…Import…第三季成果Import…importorg.apache.hadoop.mapred.lib.MultithreadedMapRunner;//LoadHadoopAPIClassMapRed{

ClassMapper…{//MapCode

}

ClassReducer…{//ReduceCode } main(){

JobConfJob=newJobConf(“MapRed.class”); Job.setMapperClass(Mapper.class); Job.setReduceClass(Reducer.class);

FileInputFormat.setInputPaths(Job,newPath(args[0])); FileOutputFormat.setOutputPath(Job,newPath(args[1]));

conf.setMapRunnerClass(MultithreadedMapRunner.class);conf.setInt("mapred.map.multithreadedrunner.threads",10);JobClient.runJob(Job);//OtherCodeSet}}Reduce…Main…Map…Import…MultithreadedMapRunner(版本0.20.0之前)

第三季成果MultithreadedMapRunner(版本0.20.0之後)

Import…importorg.apache.hadoop.mapreduce.lib.MultithreadedMapper;//LoadHadoopAPIClassMapRed{

ClassMapperextendsMultithreadedMapper<…,…,…,…>

{//MapCode }

ClassReducer…{//ReduceCode } main(){

JobConfJob=newJobConf(“MapRed.class”); Job.setMapperClass(Mapper.class); Job.setReduceClass(Reducer.class);

FileInputFormat.setInputPaths(Job,newPath(args[0])); FileOutputFormat.setOutputPath(Job,newPath(args[1]));

TokenizerMapper.setNumberOfThreads(job,4);JobClient.runJob(Job);//OtherCodeSet}}Reduce…Main…Map…Import…第三季成果1.WordcountExample

WordCount.java主程式，將相同token的sum起來。2.AverageRatingMapReduceExample

HadoopDriver.java主程式。UserRatingMapper.javaMapper

:使用者評分。AverageValueReducer.javaReducer:針對每個使用者評分進行平均計算。

IntArrayWritable.java全域物件宣告。MultithreadExample

analysis第三季成果一般wordcountMultithread

wordcountJobConfconf=newJobConf(WordCount.class);conf.setJobName("wordcount");

conf.setOutputKeyClass(Text.class);conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(Map.class);conf.setCombinerClass(Reduce.class);conf.setReducerClass(Reduce.class);

conf.setInputFormat(TextInputFormat.class);conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf,newPath(args[0]));FileOutputFormat.setOutputPath(conf,newPath(args[1]));

JobClient.runJob(conf);JobConfconf=newJobConf(WordCount.class);conf.setJobName("wordcount");

conf.setOutputKeyClass(Text.class);conf.setOutputValueClass(IntWritable.class);

conf.setMapperClass(Map.class);conf.setCombinerClass(Reduce.class);conf.setReducerClass(Reduce.class);

conf.setMapRunnerClass(MultithreadedMapRunner.class);conf.setInt("mapred.map.multithreadedrunner.threads",10);conf.setInputFormat(TextInputFormat.class);conf.setOutputFormat(TextOutputFormat.class);

FileInputFormat.setInputPaths(conf,newPath(args[0]));FileOutputFormat.setOutputPath(conf,newPath(args[1]));

JobClient.runJob(conf);

1.

WordcountExample第三季成果(2)

Set“mapred.map.multithreadedrunner.threads”為10，資訊如下。(3)

沒有Set“mapred.map.multithreadedrunner.threads”如下。(1)

Reduce的完成23.66%對應1000File完成710FileURLInformation第三季成果2.

AverageValueReducer

Example1:1NetFlixdataset:Thread=4Totalsize=146MB一般的AverageValueReducerMultithreadAverageValueReducer第三季成果一般的AverageValueReducerMultithreadAverageValueReducer2.

AverageValueReducer

Example2:NetFlixdataset:Thread=4Totalsize=10146MB(資料個數資料大小)第三季成果2.

AverageValueReducer