实验MapReduce编程初级实践

1、实验3 MaPReduCe编程初级实践1. 实验目的1. 通过实验掌握基本的 MaPRedUCe编程方法；2. 掌握用MaPRedUCe解决一些常见的数据处理问题，包括数据去重、数据排序和数据挖掘等。2. 实验平台已经配置完成的 HadooP伪分布式环境。3. 实验内容和要求1. 编程实现文件合并和去重操作对于两个输入文件，即文件A和文件B,请编写MaPRedUCe程序，对两个文件进行合并, 并剔除其中重复的内容，得到一个新的输出文件CO下面是输入文件和输出文件的一个样例供参考。实验最终结果（合并的文件）：園J*V" TEI 4MeiigejdVdQl IldriId h UOOO/

2、u Sef_ hdf5IoU IhOSt9000ua20150101y20)010)X20150IMy20150102y20150103X20150103耳20150104Z20150104y20150105y20150105ZX代码如下:PaCkage ;importClass Merge PUbIiC StatiC class MaP exte nds MaPPer<Object, Text, Text, Text> PriVate StatiC TeXt text = new Text();PUbIiC void map(Object key, TeXt value, Con

3、 text Con text) throws IOEXCePtiO n, In terruptedExcepti On text = value;(text ,new Text("");PUbIiC StatiC class RedUCe exte nds RedUCer<Text, Text, Text, Text> PUbIiC void reduce(Text key, Iterable<Text> Values, Con text Con text) throws IOEXCePtiO n, In terruptedExcepti On (k

4、ey, new Text("");PUbliC StatiC Void main( Stri ng args) throws EXCePti On Con figurati On Conf = new Con figurati on();("", "");Strin g OtherArgS = new Stri ng "in put", "output" ; if != 2) "Usage: Merge and duplicate removal <in> <out&

5、gt;"); (2);Job job = (conf, "Merge and duplicate removal");(job, new Path(OtherArgS0);(job, new Path(OtherArgS1); (true) ? 0 : 1);2. 编写程序实现对输入文件的排序现在有多个输入文件， 每个文件中的每行内容均为一个整数。要求读取所有文件中的整数，进行升序排序后，输出到一个新的文件中，输出的数据格式为每行两个整数，第一个数字为第二个整数的排序位次，第二个整数为原待排列的整数。下面是输入文件和输出文件的 一个样例供参考。实验结果截图:代码如

6、下：PaCkage ;importClass MergeSOrt PUbIiC StatiC Class MaP exte ndsMaPPer<ObjeCt, Text, IntWritable, IntWritable> private static IntWritable data = new IntWritable();public void map(Object key, Text value, Context context) throws IOException, InterruptedException String line = ();(line);(data, n

7、ew IntWritable(1);public static class Reduce extendsReducer<IntWritable, IntWritable, IntWritable, IntWritable> private static IntWritable linenum = new IntWritable(1);public void reduce(IntWritable key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedExcept

8、ion for (IntWritable val : values) (linenum, key);linenum = new IntWritable() + 1);public static void main(String args) throws Exception Configuration conf = new Configuration();("", "");直接设置输入参数String otherArgs = new String "input2", "output2" ; /* */if != 2)

9、 "Usage: mergesort <in> <out>");(2);Job job = (conf, "mergesort");(job, new Path(otherArgs0);(job, new Path(otherArgs1);(true) ? 0 : 1);3. 对给定的表格进行信息挖掘下面给出一个 child-parent 的表格，要求挖掘其中的父子辈关系，给出祖孙辈关系的 表格。实验最后结果截图如下：、伸 O R B hdfs:/IOCaIhOSt:900Jji STjOin.java hdfs:/hcalhos

10、t:900 E3 % 一 grand_Child grandjsaretMark JesseMark AliCePhiIiPJeSSePhniP AhCeJone JeSSeJQne AliCeSteVen JesseSteven AlICeSteVen FrankSteVen M aryJone Frank Jone Mary代码如下：PaCkage ;import .*;importclass STjOi n PUbIiC StatiC int time = 0;PUbIiC StatiC class MaP exte nds MaPPer<Object, Text, Text, T

11、ext> PUbIiC void map(Object key, TeXt value, Con text Con text) throws IOEXCePti on. In terruptedExcepti On Stri ng child_ name = new Stri ng();Stri ng Pare nt_n ame = new Stri ng();String relati On _type = new Strin g();Stri ng Iine =();int i = 0;while (i) != ' ') i+;Stri ng values = (0,

12、 i), (i + 1) ;if (VaIUeS0PareTo("child") != 0) child_ name = VaIUeS0; Pare nt_n ame = VaIUeS1; relati On _type = "1"(new TeXt(Values1), new Text(relati On _type + "+" + child_ name + "+" + Pare nt_n ame);relati On _type = "2"(new TeXt(Values0), new T

13、ext(relatio n_type + "+" + child_ name + "+" + Pare nt_n ame); 一 一PUbIiC StatiC class RedUCe exte nds RedUCer<Text, Text, Text, Text> PUbIiC void reduce(Text key, Iterable<Text> Values, Con text Con text) throws IOEXCePti on, In terruptedExcepti On if (time = 0) (new

14、Text("gra nd_child"), new Text("gra nd_pare nt"); time+;int gra nd_child_ num = 0;Stri ng gran d_child = new Strin g10;int grand_parent_num = 0;String grand_parent = new String10;Iterator ite = ();while () String record = ().toString();int len = ();int i = 2;if (len = 0)continue;

15、char relation_type = (0);String child_name = new String();String parent_name = new String(); while (i) != '+') child_name = child_name + (i); i+;i = i + 1;while (i < len) parent_name = parent_name + (i); i+;if (relation_type = '1') grand_childgrand_child_num = child_name; grand_ch

16、ild_num+; else grand_parentgrand_parent_num = parent_name; grand_parent_num+;if (grand_parent_num != 0 && grand_child_num != 0) for (int m = 0; m < grand_child_num; m+) for (int n = 0; n < grand_parent_num; n+) (new Text(grand_childm), new Text( grand_parentn);public static void main(S

17、tring args) throws Exception Configuration conf = new Configuration();("", "");String otherArgs = new String "input3", "output3" ;if != 2) "Usage: Single Table Join <in> <out>");(2);Job job = (conf, "Single table join ");(job, new Path(otherArgs0);(job, new Path(otherArgs1);(true) ? 0 : 1);4. 实验报告云计算实验报告题目MaP RedUCe 编程姓名包生友日期：初级实践2016/12/20实验环境：机房的虚拟机上配置好的环境解决问题的思路：根据老师给的代码进行操作实验内容与完成情况：已完成，与