大数据基础课程设计报告

1、.大数据基础课程设计报告一、项目简介 ：使用 hadoop 中的 hive 、mapreduce 以及 HBASE 对网上的一个搜狗五百万的数进行了一个比较实际的数据分析。搜狗五百万数据，是经过处理后的搜狗搜索引擎生产数据，具有真实性 ，大数据性，能够较好的满足分布式计算应用开发课程设计的数据要求。搜狗数据的数据格式为 ：访问时间 t 用户 IDt 查询词 t 该 URL 在返回结果中的排名 t 用户点击的顺序号 t 用户点击的URL。 其中 ，用户 ID 是根据用户使用浏览器访问搜索引擎时的Cookie信息自动赋值 ，即同一次使用浏览器输入的不同查询对应同一个用户ID。.专业 .专注.二、操

2、作要求1.将原始数据加载到HDFS 平台 。2.将原始数据中的时间字段拆分并拼接，添加年、月、日、小时字段 。3.将处理后的数据加载到HDFS 平台 。4.以下操作分别通过MR 和 Hive 实现 。查询总条数非空查询条数无重复总条数独立 UID 总数查询频度排名 （频度最高的前 50 词）查询次数大于 2 次的用户总数查询次数大于 2 次的用户占比Rank 在 10 以内的点击次数占比直接输入 URL 查询的比例查询搜索过 ”仙剑奇侠传 “的 uid ，并且次数大于 35.将 4 每步骤生成的结果保存到HDFS 中 。6.将 5 生成的文件通过Java API 方式导入到 HBase（一张表

3、）。7.通过 HBase shell 命令查询 6 导出的结果 。三、实验流程.专业 .专注.1. 将原始数据加载到 HDFS 平台2. 将原始数据中的时间字段拆分并拼接，添加年 、月、日、小时字段（ 1）编写 1 个脚本 sogou-log-extend.sh，其中 sogou-log-extend.sh的内容为：#!/bin/bashinfile=$1outfile=$2awk-F't''print$0"t"substr($1,0,4)"年 t"substr($1,5,2)"月 t"substr($1,7,

4、2)"日t"substr($1,8,2)"hour"' $infile > $outfile.专业 .专注.处理脚本文件 ：结果为：3. 将处理后的数据加载到 HDFS 平台.专业 .专注.4. 以下操作分别通过 MR 和 Hive 实现 .hive 实现1.查看数据库 ：show databases;2.创建数据库 : create database sogou;3.使用数据库 : use sogou;4.查看所有表 ：show tables;5.创建 sougou 表：Create table sogou(time string,uui

5、d string,namestring,num1 int,num2 int,url string) Row format delimited fieldsterminated by 't'6.将本地数据导入到 Hive 表里：Load data local inpath'/root/sogou.500w.utf8' into table sogou;7.查看表信息 ：desc sogou;.专业 .专注.（1）查询总条数select count(*) from sogou;（2）非空查询条数select count(*) from sogou where nam

6、e is not null and name !=''（3）无重复总条数select count(*) from (select * from sogou group by.专业 .专注.time,num1,num2,uuid,name,url having count(*)=1) a;（4）独立 UID 总数select count(distinct uuid) from sogou;.专业 .专注.（5）查询频度排名 （频度最高的前50 词）select name,count(*) as pd from sogou group by name order by pd des

7、climit 50;（6）查询次数大于2 次的用户总数select count(a.uuid) from (select uuid,count(*) as cnt from sogou groupby uuid having cnt > 2) a;.专业 .专注.（7）查询次数大于2 次的用户占比select count(*) from (select uuid,count(*) as cnt from sogou group byuuid having cnt > 2) a;.专业 .专注.（8） Rank 在 10 以内的点击次数占比select count(*) from s

8、ogou where num1<11;（9）直接输入 URL 查询的比例select count(*) from sogou where url like '%www%'（10 ）查询搜索过 ”仙剑奇侠传 “的 uid ，并且次数大于3select uuid ,count(*) as uu from sogou where name='仙剑奇侠传 ' group.专业 .专注.by uuid having uu>3;.MapReduce实现 (import的各种包省略 )（1）查询总条数public class MRCountAll public s

9、tatic Integer i = 0;public static boolean flag = true;public static class CountAllMap extends Mapper<Object, Text, Text, Text> Overrideprotectedvoidmap(Objectkey,Textvalue,Mapper<Object,Text,Text,Text>.Context context)throws IOException, InterruptedException i+;.专业 .专注.public static void

10、runcount(String Inputpath, String Outpath) Configuration conf = new Configuration();Job job = null;try job = Job.getInstance(conf, "count"); catch (IOException e) / TODO Auto-generated catch blocke.printStackTrace();job.setJarByClass(MRCountAll.class);job.setMapperClass(CountAllMap.class);

11、job.setOutputKeyClass(Text.class);job.setOutputValueClass(Text.class);try FileInputFormat.addInputPath(job, new Path(Inputpath); catch (IllegalArgumentException e) / TODO Auto-generated catch blocke.printStackTrace(); catch (IOException e) / TODO Auto-generated catch block e.printStackTrace();.专业 .专

12、注.FileOutputFormat.setOutputPath(job, new Path(Outpath);try job.waitForCompletion(true); catch (ClassNotFoundException e) / TODO Auto-generated catch block e.printStackTrace(); catch (IOException e) / TODO Auto-generated catch block e.printStackTrace(); catch (InterruptedException e) / TODO Auto-gen

13、erated catch block e.printStackTrace();public static void main(String args) throws Exception 总条数 :" + i);.专业 .专注.（2）非空查询条数public class CountNotNull public static String Str = ""public static int i = 0;public static boolean flag = true;public static class wyMap extends Mapper<Object

14、, Text, Text, IntWritable> Overrideprotectedvoidmap(Objectkey,Textvalue,Mapper<Object,Text,Text,IntWritable>.Context context)throws IOException, InterruptedException String values = value.toString().split("t");if (!values2.equals(null) && values2 != "") context.w

15、rite(new Text(values1), new IntWritable(1);i+;public static void run(String inputPath, String outputPath) Configuration conf = new Configuration();Job job = null;try .专业 .专注.job = Job.getInstance(conf, "countnotnull"); catch (IOException e) / TODO Auto-generated catch blocke.printStackTrac

16、e();assert job != null;job.setJarByClass(CountNotNull.class);job.setMapperClass(wyMap.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);try FileInputFormat.addInputPath(job, new Path(inputPath); catch (IllegalArgumentException e) e.printStackTrace(); catch (IOExcept

17、ion e) e.printStackTrace();try FileOutputFormat.setOutputPath(job, new Path(outputPath);job.waitForCompletion(true); catch (ClassNotFoundException e) e.printStackTrace();.专业 .专注. catch (IOException e) e.printStackTrace(); catch (InterruptedException e) e.printStackTrace();public static void main(Str

18、ing args) 非空条数 :" + i);（3）无重复总条数public class CountNotRepeat public static int i = 0;public static class NotRepeatMap extends Mapper<Object , Text , Text, Text>Overrideprotectedvoidmap(Objectkey,Textvalue,Mapper<Object,Text,Text,Text>.Context context) throws IOException, InterruptedE

19、xception String text = value.toString();String values = text.split("t");.专业 .专注.String time = values0;String uid = values1;String name = values2;String url = values5;context.write(new Text(time+uid+name+url), new Text("1");publicstaticclassNotRepeatReducextendsReducer<Text,Int

20、Writable,Text,IntWritable>Overrideprotectedvoidreduce(Textkey,Iterable<IntWritable>values,Reducer<Text,IntWritable,Text,IntWritable>.Contextcontext)throwsIOException,InterruptedException i+;context.write(new Text(key.toString(),new IntWritable(i);public static void main(String args) t

21、hrows IOException, ClassNotFoundException,InterruptedException Configuration conf = new Configuration();Job job = null;.专业 .专注.try job = Job.getInstance(conf, "countnotnull"); catch (IOException e) / TODO Auto-generated catch block e.printStackTrace();assert job != null;job.setJarByClass(C

22、ountNotRepeat.class);job.setMapperClass(NotRepeatMap.class);job.setReducerClass(NotRepeatReduc.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(Text.class);try FileInputFormat.addInputPath(job,new catch (IllegalArgumentException e) e.printStackTrace(); catch (IOException e) e.printSt

23、ackTrace();try FileOutputFormat.setOutputPath(job,new.专业 .专注.Path("/sogou/data/CountNotRepeat");job.waitForCompletion(true); catch (ClassNotFoundException e) e.printStackTrace(); catch (IOException e) e.printStackTrace(); catch (InterruptedException e) e.printStackTrace();无重复总条数为:" +

24、i);（4）独立 UID 总数public class CountNotMoreUid public static int i = 0;public static class UidMap extends Mapper<Object , Text , Text, Text>Overrideprotectedvoidmap(Objectkey,Textvalue,Mapper<Object,Text,Text,Text>.Context context) throws IOException, InterruptedException String text = valu

25、e.toString();.专业 .专注.String values = text.split("t");String uid = values1;context.write(new Text(uid), new Text("1");public static class UidReduc extends Reducer<Text , IntWritable, Text, IntWritable>Overrideprotectedvoidreduce(Textkey,Iterable<IntWritable>values,Redu

26、cer<Text,IntWritable,Text,IntWritable>.Contextcontext)throwsIOException,InterruptedException i+;context.write(new Text(key.toString(),new IntWritable(i);public static void main(String args) throws IOException, ClassNotFoundException,InterruptedException Configuration conf = new Configuration()

27、;Job job = null;try job = Job.getInstance(conf, "countnotnull");.专业 .专注. catch (IOException e) / TODO Auto-generated catch block e.printStackTrace();assert job != null;job.setJarByClass(CountNotNull.class);job.setMapperClass(UidMap.class);job.setReducerClass(UidReduc.class);job.setOutputKe

28、yClass(Text.class);job.setOutputValueClass(Text.class);try FileInputFormat.addInputPath(job,new catch (IllegalArgumentException e) e.printStackTrace(); catch (IOException e) e.printStackTrace();try FileOutputFormat.setOutputPath(job,newPath("/sogou/data/CountNotMoreUid");job.waitForComplet

29、ion(true);.专业 .专注. catch (ClassNotFoundException e) e.printStackTrace(); catch (IOException e) e.printStackTrace(); catch (InterruptedException e) e.printStackTrace();独立 UID 条数 :" + i);（5）查询频度排名 （频度最高的前50 词）public class CountTop50 publicstaticclassTopMapperextendsMapper<LongWritable,Text,Tex

30、t,LongWritable>Text text =new Text();Overrideprotected void map(LongWritable key, Text value,Context context)throws IOException, InterruptedException String line= value.toString().split("t");String keys = line2;text.set(keys);.专业 .专注.context.write(text,new LongWritable(1);publicstaticcl

31、assTopReducerextendsReducer<Text,LongWritable,Text,LongWritable>Text text = new Text();TreeMap<Integer,String > map = new TreeMap<Integer,String>();Overrideprotected void reduce(Text key, Iterable<LongWritable> value, Context context)throws IOException, InterruptedException i

32、nt sum=0;/key出现次数for (LongWritable ltext : value) sum+=ltext.get();map.put(sum,key.toString();/ 去前 50 条数据if(map.size()>50)map.remove(map.firstKey();Overrideprotected void cleanup(Context context).专业 .专注.throws IOException, InterruptedException for(Integer count:map.keySet()context.write(new Text(

33、map.get(count), new LongWritable(count);public static void main(String args) throws IOException, ClassNotFoundException,InterruptedException Configuration conf = new Configuration();Job job = Job.getInstance(conf, "count");job.setJarByClass(CountTop50.class);job.setJobName("Five"

34、);job.setOutputKeyClass(Text.class);job.setOutputValueClass(LongWritable.class);job.setMapperClass(TopMapper.class);job.setReducerClass(T opReducer.class);FileInputFormat.addInputPath(FileOutputFormat.setOutputPath(job, new Path("/sogou/data/CountTop50");job.waitForCompletion(true);.专业 .专注

35、.（6）查询次数大于2 次的用户总数public class CountQueriesGreater2 public static int total = 0;public static class MyMaper extends Mapper<Object, Text, Text, IntWritable> protectedvoidmap(Objectkey,Textvalue,Mapper<Object,Text,Text,IntWritable>.Context context)throws IOException, InterruptedException S

36、tring str = value.toString().split("t");Text word;IntWritable one = new IntWritable(1);word = new Text(str1);context.write(word, one);public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable>Overrideprotected void reduce(Text arg0, Iterable<IntWritable>

37、; arg1,Reducer<Text,IntWritable,Text, IntWritable>.Contextarg2) throws IOException, InterruptedException / arg0 是一个单词 arg1 是对应的次数int sum = 0;.专业 .专注.for (IntWritable i : arg1) sum += i.get();if(sum>2)total=total+1;/arg2.write(arg0, new IntWritable(sum);public static void main(String args) t

38、hrows IOException, ClassNotFoundException,InterruptedException Configuration conf = new Configuration();/ 1. 实例化一个JobJob job = Job.getInstance(conf, "six");/ 2. 设置 mapper类job.setMapperClass(MyMaper.class);/ 3. 设置 Combiner 类 不是必须的/ job.setCombinerClass(MyReducer.class);/ 4. 设置 Reducer 类job.

39、setReducerClass(MyReducer.class);/ 5. 设置输出key 的数据类型.专业 .专注.job.setOutputKeyClass(Text.class);/ 6. 设置输出 value 的数据类型job.setOutputValueClass(IntWritable.class);/设置通过哪个类查找job 的 Jar 包job.setJarByClass(CountQueriesGreater2.class);/ 7. 设置输入路径FileInputFormat.addInputPath(/ 8. 设置输出路径FileOutputFormat.setOutpu

40、tPath(job,newPath("/sogou/data/CountQueriesGreater2");/ 9. 执行该作业job.waitForCompletion(true);查询次数大于2 次的用户总数： " + total + "条 ");（7）查询次数大于2 次的用户占比public class CountQueriesGreaterPro public static int total1 = 0;public static int total2 = 0;public static class MyMaper extends Ma

41、pper<Object, Text, Text, IntWritable> Override.专业 .专注.protectedvoidmap(Objectkey,Textvalue,Mapper<Object,Text,Text,IntWritable>.Context context)throws IOException, InterruptedException total2+;String str = value.toString().split("t");Text word;IntWritable one = new IntWritable(1);word = new Text(str1);context.write(word, one);/执行完毕后就是一个单词对应一个value(1)public static class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable>Overrideprotected void reduce(Text a