dataX3.0安装使用手册

1、【精品文档】如有侵权，请联系网站删除，仅供学习与交流dataX3.0安装使用手册.精品文档.DataX3.0使用手册目录一、dataX概述11dataX作用12DataX3.0框架设计23DataX3.0插件体系2二、dataX安装31创建用户组及用户32环境变量配置43安装python4三、dataX使用41生成配置文件模板42配置定时任务73常见问题及处理8四、dataX插件开发及安装81.Reader插件开发82.Writter插件开发93.插件部署10一、 dataX概述1 dataX作用DataX 是一个异构数据源离线同步工具，致力于实现包括关系型数据库(MySQL、Oracle等)

2、、HDFS、Hive、ODPS、HBase、FTP等各种异构数据源之间稳定高效的数据同步功能。为了解决异构数据源同步问题，DataX将复杂的网状的同步链路变成了星型数据链路，DataX作为中间传输载体负责连接各种数据源。当需要接入一个新的数据源的时候，只需要将此数据源对接到DataX，便能跟已有的数据源做到无缝数据同步。2 DataX3.0框架设计DataX本身作为离线数据同步框架，采用Framework + plugin架构构建。将数据源读取和写入抽象成为Reader/Writer插件，纳入到整个同步框架中。Reader：Reader为数据采集模块，负责采集数据源的数据，将数据发送给Fram

3、ework。Writer： Writer为数据写入模块，负责不断向Framework取数据，并将数据写入到目的端。Framework：Framework用于连接reader和writer，作为两者的数据传输通道，并处理缓冲，流控，并发，数据转换等核心技术问题。3 DataX3.0插件体系类型数据源Reader(读)Writer(写)RDBMS 关系型数据库MySQL Oracle SQL Server PostgreSQL 达梦 通用RDBMS(支持所有关系型数据库)阿里云数仓数据存储MaxCompute(原ODPS) Analyti

4、c DB(原ADS)  OSS 云数据库Memcache版(原OCS)HiveNoSQL数据存储Table Store(原OTS) Hbase0.94 Hbase1.1 MongoDB无结构化数据存储TxtFileJsonFile FTP HDFS二、 dataX安装1 创建用户组及用户roothmaster-hdfs:/home/ubuntu# groupadd dataxroothmaster-hdfs:/home/ubuntu# useradd -g datax datax -m -d /home/datax

5、roothmaster-hdfs:/home/ubuntu# passwd dataxEnter new UNIX password: Retype new UNIX password: passwd: password updated successfullyroothmaster-hdfs:/home/ubuntu# usermod -G adm -a dataxroothmaster-hdfs:/home/ubuntu# su - dataxdataxhmaster-hdfs:$ iduid=1004(datax) gid=1000(ubuntu) groups=1000(ubuntu)

6、,4(adm),1002(datax)2 环境变量配置export JAVA_HOME=/usr/java/jdk1.7.0_80export JRE_HOME=$JAVA_HOME/jreexport CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$JAVA_HOME/binexport LANG=zh_CN.UTF-83 安装pythondataxhmaster-hdfs:/home/ubuntu$ pythonPython 2.7.6 (default, Jun 22 2015, 17:58:13) GCC 4.8.2 on li

7、nux2Type "help", "copyright", "credits" or "license" for more information.三、 dataX使用1 生成配置文件模板命令：bin/python datax.py -r YOUR_READER -w YOUR_WRITER案例1：mysqlreader-> mysqlwriterdataxhmaster-hdfs:/datax3/datax/bin$ python datax.py -r mysqlreader -w mysqlwriter

8、DataX (DATAX-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Please refer to the mysqlreader document: Please refer to the mysqlwriter document: Please save the following configuration as a json file and use python DATAX_HOME/bin/datax.py JSON_FILE_NAME.jso

9、n to run the job. "job": "content": "reader": "name": "mysqlreader", "parameter": "column": , "connection": "jdbcUrl": , "table": , "password": "", "username": "&quo

10、t;, "where": "" , "writer": "name": "mysqlwriter", "parameter": "column": , "connection": "jdbcUrl": "", "table": , "password": "", "preSql": , "session&qu

11、ot;: , "username": "", "writeMode": "" , "setting": "speed": "channel": "" 案例2：jsonfilereader-> hbasewriterdataxhmaster-hdfs:/datax3/datax/bin$ python datax.py -r jsonfilereader -w hbasewriter DataX (DATAX-OPENSOURCE-

12、1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Please refer to the jsonfilereader document: Please refer to the hbasewriter document: Please save the following configuration as a json file and use python DATAX_HOME/bin/datax.py JSON_FILE_NAME.json to run the job. &qu

13、ot;job": "content": "reader": "name": "jsonfilereader", "parameter": "column": , "compress": "zip", "encoding": "", "path": , "writer": "name": "hbase11xwriter&

14、quot;, "parameter": "column": , "encoding": "", "hbaseConfig": "hbase.cluster.distributed": "", "hbase.rootdir": "", "hbase.zookeeper.quorum": "" , "mode": "", "rowk

15、eyColumn": , "table": "", "versionColumn": "index": "", "value": "" , "setting": "speed": "channel": "" 2 配置定时任务1.1 创建任务列表文件dataxhmaster-hdfs:/datax3/datax/task$ vi crontab.tasks.txt 20

16、,30,40 0-23 * * * /usr/bin/python /home/datax/datax3/datax/bin/datax.py /home/datax/datax3/datax/job/jsonfile2hbase_fullry_log.terminal_log.json >/data/datax/log/datax.crontab.out.log 2>&11.2 crontab加载系统任务列表文件dataxhmaster-hdfs:/datax3/datax/task$ crontab crontab.tasks.txt dataxhmaster-hdfs

17、:/datax3/datax/task$ crontab -l20,30,40 0-23 * * * /usr/bin/python /home/datax/datax3/datax/bin/datax.py /home/datax/datax3/datax/job/jsonfile2hbase_fullry_log.terminal_log.json >/data/datax/log/datax.crontab.out.log 2>&13 常见问题及处理1.1 /bin/sh: 1: java: not found原因：/home/datax/datax3/datax/b

18、in/ datax.py未取到JAVA_HOME环境变量值四、 dataX插件开发及安装1. Reader插件开发public class JsonFileReader extends Readerpublic static class Job extends Reader.Job Overridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic List<Configuratio

19、n> split(int adviceNumber) / TODO Auto-generated method stubreturn null;public static class Task extends Reader.TaskOverridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic void startRead(RecordSender recordSender) /

20、 TODO Auto-generated method stub2. Writter插件开发public class JsonFileWriter extends Writer public static class Job extends Writer.Job Overridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic List<Configuration> spli

21、t(int mandatoryNumber) / TODO Auto-generated method stubreturn null;public static class Task extends Writer.Task Overridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic void startWrite(RecordReceiver lineReceiver) / TO

22、DO Auto-generated method stub3. 插件部署目录及文件说明dataxhmaster-hdfs:/datax3/datax/plugin/reader/jsonfilereader$ lltotal 56drwxr-xr-x 3 datax ubuntu 4096 Jul 6 16:10 ./drwxr-xr-x 21 datax ubuntu 4096 Jun 21 11:05 ./-rw-r-r- 1 datax ubuntu 36348 Jun 22 17:52 jsonfilereader.jardrwxr-xr-x 2 datax ubuntu 4096 May 16 12:46 libs/-rw- 1 datax ubuntu 315 May 16 11:24 plugin.json-rw- 1 datax ubuntu 149 May 15 18:24 plugin_job_template.jsonl jsonfilereader.jar 插件J