大数据采集与清洗PPT精选文档

1、.1 数据采集与清洗数据采集与清洗 2019|02|15 周 乐.2什么是大数据大数据处理流程大数据的主要特征大数据采集的概念大数据采集应用.31什么是大数据.4.5.6.7淘宝推荐 依据购物行为偏好依据购物行为偏好引荐引荐依据你最近的阅读依据你最近的阅读行为和消费行为进行为和消费行为进行引荐行引荐依据你用的设备往依据你用的设备往来不断猜特征来不断猜特征. .依据时节改变进行依据时节改变进行引荐引荐.82014-032015-082017-102016-032018行业现状与前景行业现状与前景.9.102019年人社部拟最新发布15项新职业1.大数据工程技术人员大数据工程技术人员2.云计算工程

2、技术人员云计算工程技术人员3.人工智能工程技术人员人工智能工程技术人员4.物联网工程技术人员物联网工程技术人员5.11.12.13什么是大数据大数据（大数据（Big DataBig Data）是指无法使用是指无法使用传统和常用的软件技术和工具在一定时传统和常用的软件技术和工具在一定时间内完成获取、管理和处理的数据集间内完成获取、管理和处理的数据集.142大数据的主要特征.15大数据主要特征 VolumeVolumeVelocityVelocityVarietyVarietyVeracityVeracity真实性（真实性（VeracityVeracity），），即追求高质量的数即追求高质量的数据

3、。据。容量大（容量大（VolumeVolume）, ,指大规模的数据量，指大规模的数据量，并且数据量呈持续并且数据量呈持续增长趋势。增长趋势。速度快（速度快（VelocityVelocity）, ,指的是数据被创建指的是数据被创建和移动的速度。和移动的速度。种类多（种类多（VarietyVariety）, ,指数据来自多种数指数据来自多种数据源，数据种类和据源，数据种类和格式。格式。ValueValue价值密度低价值密度低（ValueValue），指随着），指随着数据量的增长，数数据量的增长，数据中有意义的信息据中有意义的信息却没有成相应比例却没有成相应比例增长。增长。.163大数据处理流程.

4、17大数据处理流程 数据预处理数据预处理 就是将采集就是将采集来的数据从多种数据库来的数据从多种数据库导入到大型的分布式数导入到大型的分布式数据库中（目前主要是据库中（目前主要是hfdshfds或或hivehive）, ,并同时做并同时做一些简单的清洗和预处一些简单的清洗和预处理工作。理工作。数据统计分析数据统计分析 就是对上面就是对上面已经完成的存储在大型分已经完成的存储在大型分布式数据库中的数据进行布式数据库中的数据进行归类统计，可以满足一般归类统计，可以满足一般场景的分析需求。场景的分析需求。数据挖掘数据挖掘 是对数据进是对数据进行基于各种算法的分析行基于各种算法的分析计算，从而起到预测

5、的计算，从而起到预测的效果，实现一些高级别效果，实现一些高级别数据分析的需求。数据分析的需求。数据采集数据采集 就是利用就是利用多种数据库（关系型，多种数据库（关系型，NOSQLNOSQL）去存储不）去存储不同来源的数据。同来源的数据。数据展示数据展示 就是对就是对以上处理完的结果以上处理完的结果进行分析，或者形进行分析，或者形成报表。成报表。.184大数据采集的概念.19大数据采集的概念3 3、大数据采集技术方法、大数据采集技术方法 大数据采集技术就是对数据进行大数据采集技术就是对数据进行 ETL ETL 操作，通过对数据进行提取、转换、加载，最操作，通过对数据进行提取、转换、加载，最终挖掘

6、数据的潜在价值。终挖掘数据的潜在价值。ETLETL指的是指的是Extract-Transform-LoadExtract-Transform-Load，也就是抽取、转换、加，也就是抽取、转换、加载。载。 抽取抽取-从各种数据源获取数据从各种数据源获取数据 转换转换-按需求格式将源数据转换为目标数据按需求格式将源数据转换为目标数据 加载加载-把目标数据加载到数据仓库中把目标数据加载到数据仓库中2 2、数据采集与大数据采集的区别、数据采集与大数据采集的区别 传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库

7、大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库1 1、什么是数据采集、什么是数据采集 数据采集就是数据获取，数据源主要分为线上数据和内容数据数据采集就是数据获取，数据源主要分为线上数据和内容数据.20大数据采集系统1.日志采集系统（Apache Flume、Scribe）3.数据库采集系统（关系型、nosql等各种数据库）2.网络数据采集系统（Scrapy 框架、Apache Nutch）.215大数据采集应用.22.23技能准备PythonPython基础基础LinuxLinux操作系统基本操作操作系统基本操作

8、数据库基础（数据库基础（SQLSQL语句操作）语句操作）.24环境准备PythonPythonJdk(javaJdk(java环境环境) )数据库（数据库（mysqlmysql）.25Thanks .26YOUR TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing

9、 is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to th

10、e man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.27YOUR TITLE Nothing is difficult to the man who Nothing is difficu

11、lt to the man who will try.Nothing is difficult to the will try.Nothing is difficult to the man who will try.Nothing is difficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who wil

12、l try.Nothing is difficult to the man who Nothing is difficult to the man who will try.Nothing is difficult to the will try.Nothing is difficult to the man who will try.Nothing is difficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is di

13、fficult to the man who will try.difficult to the man who will try.282OKPPT工作室.29YOUR TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing

14、is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the

15、 man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.30YOUR TITLE Nothing is difficult to the Nothing is difficult to the man who will try.Not

16、hing man who will try.Nothing is difficult to the man is difficult to the man who will try.who will try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.Nothing is diffi

17、cult to Nothing is difficult to the man who will the man who will try.Nothing is difficult try.Nothing is difficult to the man who will to the man who will try.try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man is dif

18、ficult to the man who will try.who will try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.31YOUR TITLE 21%9%28%42%.323OKPPT工作室.33YOUR TITLE Nothing is difficult to th

19、e man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is diff

20、icult to the man who will try.Nothing is difficult to the man who will try. try.Nothing is difficult to the man who will try. Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will tr

21、y.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.34YOUR TITLE Nothing is

22、difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will

23、try.Nothing is difficult to the man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.35YOUR TITLE Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who tr