大数据采集与清洗

1、 数据采集与清洗数据采集与清洗 2019|02|15 周 乐什么是大数据大数据处理流程大数据的主要特征大数据采集的概念大数据采集应用1什么是大数据淘宝推荐 依据购物行为偏好依据购物行为偏好引荐引荐依据你最近的阅读依据你最近的阅读行为和消费行为进行为和消费行为进行引荐行引荐依据你用的设备往依据你用的设备往来不断猜特征来不断猜特征. .依据时节改变进行依据时节改变进行引荐引荐2014-032015-082017-102016-032018行业现状与前景2019年人社部拟最新发布15项新职业1.大数据工程技术人员大数据工程技术人员2.云计算工程技术人员云计算工程技术人员3.人工智能工程技术人员人工智

2、能工程技术人员4.物联网工程技术人员物联网工程技术人员5.什么是大数据大数据（大数据（Big DataBig Data）是指无法使用是指无法使用传统和常用的软件技术和工具在一定时传统和常用的软件技术和工具在一定时间内完成获取、管理和处理的数据集间内完成获取、管理和处理的数据集2大数据的主要特征大数据主要特征 VolumeVolumeVelocityVelocityVarietyVarietyVeracityVeracity真实性（真实性（VeracityVeracity），），即追求高质量的数即追求高质量的数据。据。容量大（容量大（VolumeVolume）, ,指大规模的数据量，指大规模的数

3、据量，并且数据量呈持续并且数据量呈持续增长趋势。增长趋势。速度快（速度快（VelocityVelocity）, ,指的是数据被创建指的是数据被创建和移动的速度。和移动的速度。种类多（种类多（VarietyVariety）, ,指数据来自多种数指数据来自多种数据源，数据种类和据源，数据种类和格式。格式。ValueValue价值密度低价值密度低（ValueValue），指随着），指随着数据量的增长，数数据量的增长，数据中有意义的信息据中有意义的信息却没有成相应比例却没有成相应比例增长。增长。3大数据处理流程大数据处理流程 数据预处理数据预处理 就是将采集就是将采集来的数据从多种数据库来的数据从多种

4、数据库导入到大型的分布式数导入到大型的分布式数据库中（目前主要是据库中（目前主要是hfdshfds或或hivehive）, ,并同时做并同时做一些简单的清洗和预处一些简单的清洗和预处理工作。理工作。数据统计分析数据统计分析 就是对上面就是对上面已经完成的存储在大型分已经完成的存储在大型分布式数据库中的数据进行布式数据库中的数据进行归类统计，可以满足一般归类统计，可以满足一般场景的分析需求。场景的分析需求。数据挖掘数据挖掘 是对数据进是对数据进行基于各种算法的分析行基于各种算法的分析计算，从而起到预测的计算，从而起到预测的效果，实现一些高级别效果，实现一些高级别数据分析的需求。数据分析的需求。数

5、据采集数据采集 就是利用就是利用多种数据库（关系型，多种数据库（关系型，NOSQLNOSQL）去存储不）去存储不同来源的数据。同来源的数据。数据展示数据展示 就是对就是对以上处理完的结果以上处理完的结果进行分析，或者形进行分析，或者形成报表。成报表。4大数据采集的概念大数据采集的概念3 3、大数据采集技术方法、大数据采集技术方法 大数据采集技术就是对数据进行大数据采集技术就是对数据进行 ETL ETL 操作，通过对数据进行提取、转换、加载，最操作，通过对数据进行提取、转换、加载，最终挖掘数据的潜在价值。终挖掘数据的潜在价值。ETLETL指的是指的是Extract-Transform-LoadE

6、xtract-Transform-Load，也就是抽取、转换、加，也就是抽取、转换、加载。载。 抽取抽取-从各种数据源获取数据从各种数据源获取数据 转换转换-按需求格式将源数据转换为目标数据按需求格式将源数据转换为目标数据 加载加载-把目标数据加载到数据仓库中把目标数据加载到数据仓库中2 2、数据采集与大数据采集的区别、数据采集与大数据采集的区别 传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库传统数据采集：来源单一，数据量相当小；结构单一；关系数据库和并行数据库 大数据的数据采集：来源广泛，数量巨大；数据类型丰富；分布式数据库大数据的数据采集：来源广泛，数量巨大；数据类

7、型丰富；分布式数据库1 1、什么是数据采集、什么是数据采集 数据采集就是数据获取，数据源主要分为线上数据和内容数据数据采集就是数据获取，数据源主要分为线上数据和内容数据大数据采集系统1.日志采集系统（Apache Flume、Scribe）3.数据库采集系统（关系型、nosql等各种数据库）2.网络数据采集系统（Scrapy 框架、Apache Nutch）5大数据采集应用技能准备PythonPython基础基础LinuxLinux操作系统基本操作操作系统基本操作数据库基础（数据库基础（SQLSQL语句操作）语句操作）环境准备PythonPythonJdk(javaJdk(java环境环境)

8、)数据库（数据库（mysqlmysql）Thanks YOUR TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.N

9、othing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult N

10、othing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.YOUR TITLE Nothing is difficult to the man who Nothing is difficult to the man who will try.Nothing is difficult to the will try.Nothing is di

11、fficult to the man who will try.Nothing is difficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who will try.Nothing is difficult to the man who Nothing is difficult to the man who

12、 will try.Nothing is difficult to the will try.Nothing is difficult to the man who will try.Nothing is difficult man who will try.Nothing is difficult to the man who will try.Nothing is to the man who will try.Nothing is difficult to the man who will try.difficult to the man who will try.2OKPPT工作室YO

13、UR TITLE Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is

14、 difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the man who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.Nothing is difficult Nothing is difficult to the ma

15、n who to the man who will try.Nothing is will try.Nothing is difficult to the man difficult to the man who will try.who will try.YOUR TITLE Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man is difficult to the man who wi

16、ll try.who will try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.Nothing is difficult to Nothing is difficult to the man who will the man who will try.Nothing is dif

17、ficult try.Nothing is difficult to the man who will to the man who will try.try.Nothing is difficult to the Nothing is difficult to the man who will try.Nothing man who will try.Nothing is difficult to the man is difficult to the man who will try.who will try.Nothing is difficult to the Nothing is d

18、ifficult to the man who will try.Nothing man who will try.Nothing is difficult to the man who is difficult to the man who will try.will try.YOUR TITLE 21%9%28%42%3OKPPT工作室YOUR TITLE Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man

19、who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try. try.Nothing is d

20、ifficult to the man who will try. Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will

21、try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.YOUR TITLE Nothing is difficult to the man who will try.Nothing is difficult to the man who will Nothing is difficul

22、t to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.try.Nothing is difficult to the man who will try.Nothing is difficult to the man who will try.Nothing is Nothing is difficult to the diffic

23、ult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.Nothing is Nothing is difficult to the difficult to the man who will try.man who will try.YOUR TITLE Nothing is difficult to the man who will Nothing is difficult to the man who will try.Nothing is difficult to the man who try.Noth