基于docker搭建spark on yarn及可视化桌面

基于docker大数据集群搭建技术手册一、简介2二、docker搭建31.docker搭建32.weave搭建3三、hadoop集群镜像搭建41.搭建准备42.hadoop部署43.扩展：HA部署114.扩展：namenode的手动删除11四、集群部署与启动111.docker搭建master和slave集群112.weave设置ip地址123.服务启动：12五、基于ambari管理平台的镜像搭建141.部署准备142.私库配置143.ambari-server安装144.ambari-client安装165.ambari管理17六、桌面系统XFCE搭建17七、附录18一、 简介大数据集群部署依托于docker，组建虚拟集群，机组间通过weave搭建互通网络环境，如图1：hbase3Nodejs2HadoopHadoopweave其他mongo4其他tomcat物理机32物理机31dockerdocker图1hadoop集群搭建YARN平台，基于HDFS。在YARN基础上搭建SPARK进行业务处理，如图2：图2环境版本：1) Ubuntu 14.042) jdk:1.73) docker:1.9.14) hadoop-2.6.0-cdh5.5.05) scala-2.11.76) spark-1.5.0-bin-hadoop2.6.tgz7) zookeeper-3.4.5-cdh5.5.0下载地址见附录1注：集群部署所有权限都为root，请以root权限登陆启动服务二、 docker搭建1. docker搭建# apt-get install software-properties-common # apt-get install python-software-properties# add-apt-repository ppa:dotcloud/lxc-docker # apt-get update # apt-get install lxc-docker# docker pull ubuntu:14.04# dockerrun-i-tubuntu:14.04/bin/bash2. weave搭建# sudo wget -O /usr/local/bin/weave /zettio/weave/master/weave# sudo chmod a+x /usr/local/bin/weave# weave launch三、 hadoop集群镜像搭建1. 搭建准备进入新建的一个docker实例，将上述软件包下载并解压到/opt/目录下，各软件包安装路径如下hadoop: /opt/yarn/hadoop-2.6.0-cdh5.5.0spark: /opt/spark-1.5.0-cdh5.5.0scala: /opt/scala-2.11.72. hadoop部署1) 配置jdk等环境变量:#vim /etc/profile添加如下配置(根据所安装目录修改路径)：export JAVA_HOME=/usr/local/jdkexport CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/libexport SCALA_HOME=/opt/scalaexport SPARK_HOME=/opt/sparkexport PATH=$PATH:$JAVA_HOME/bin:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/binexport HADOOP_DEV_HOME=/opt/yarn/hadoop-2.6.0-cdh5.5.0/export PATH=$PATH:$HADOOP_DEV_HOME/binexport PATH=$PATH:$HADOOP_DEV_HOME/sbinexport HADOOP_MAPARED_HOME=$HADOOP_DEV_HOMEexport HADOOP_COMMON_HOME=$HADOOP_DEV_HOMEexport HADOOP_HDFS_HOME=$HADOOP_DEV_HOMEexport YARN_HOME=$HADOOP_DEV_HOMEexport HADOOP_CONF_DIR=$HADOOP_DEV_HOME/etc/hadoopexport HDFS_CONF_DIR=$HADOOP_DEV_HOME/etc/hadoopexport YARN_CONF_DIR=$HADOOP_DEV_HOME/etc/hadoop#source profile2) 配置host和hostname localhost master slave1 slave23 slave134 slave145 slave15 3) 创建数据和日志目录 hadoop需要不同的日志目录，创建以下目录#mkdir -p /var/data/hadoop/hdfs/nn#mkdir -p /var/data/hadoop/hdfs/snn#mkdir -p /var/data/hadoop/hdfs/dn4) 在hadoop目录下建立logs目录，并设置权限#cd /opt/yarn/hadoop-2.6.0-cdh5.5.0#mkdir logs#chmod g+w logs5) 配置core-site.xml #cd /opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop添加如下配置： hdfs:/master:9000 hadoop.http.staticuser.user hdfs hadoop.tmp.dir /hadoop/tmp A base for other temporary directories. 6) 配置hdfs-site.xml同上，添加如下配置： dfs.replication 3 .dir file:/var/data/hadoop/hdfs/nn fs.checkpoint.dir file:/var/data/hadoop/hdfs/snn fs.checkpoint.edits.dir file:/var/data/hadoop/hdfs/snn dfs.datanode.data.dir file:/var/data/hadoop/hdfs/dn dfs.hosts.exclude /opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/exclude 7) 配置mapred-site.xml mapred.job.history.server.embedded true mapreduce.jobhistory.address master:10020 mapreduce.jobhistory.webapp.address master:50060 ermediate-done-dir /mr-history/tmp mapreduce.jobhistory.done-dir /mr-history/done yarn 8) 配置yarn-site.xml yarn.resourcemanager.address master:8032 yarn.resourcemanager.scheduler.address master:8030 yarn.resourcemanager.resource-tracker.address master:8035 yarn.resourcemanager.admin.address master:8033 yarn.resourcemanager.webapp.address master:8088 yarn.resourcemanager.hostname master yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler 9) 修改slaves文件slave1slave2slave13slave14slave1510) spark配置#cd /opt/spark-1.5.0-cdh5.5.0/conf修改spark-env.sh，添加如下配置：export SCALA_HOME=/opt/scalaexport JAVA_HOME=/usr/local/jdkexport HADOOP_HOME=/opt/yarn/hadoop-2.6.0-cdh5.5.0/export SPARK_MASTER_IP=masterexport SPARK_WORKER_MEMORY=1024mexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop11) 修改启动脚本#cd /bin/#touch boot.sh#chmod 777 boot.sh#vim boot.sh添加以下配置：#!/usr/bin/env bashsource /etc/profileservice ssh start/bin/bash12) 配置免登陆（实体机部署参照网上hadoop集群ssh免登陆配置）apt安装ssh并启动，并在/root/.ssh/下执行ssh-keygen生成秘钥将id_rsa.pub文件内容添加到.ssh/authorized_keys13) 挂载文件（实体机部署略过）退出系统，将上述修改过的配置文件挂载到docker容器下，方便修改#cd #mkdir env#cd env#touch hosts#mkdir hadoop#mkdir spark拉取相应的文件14) 将配置好的单机做成镜像#docker commit -m=spark -author=IEDS a790e8142381 ieds/ spark-base:v63. 扩展：HA部署4. 扩展：namenode的手动删除四、 集群部署与启动1. docker搭建master和slave集群#docker run -idt -p 50070:50070 -p 8088:8088 -p 4040:4040 -p 50075:50075 -p 8080:8080 -v /env/hosts:/etc/hosts -v /env/hadoop/hadoop-env.sh:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/hadoop-env.sh -v /env/hadoop/slaves:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/slaves -v /env/hadoop/core-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/core-site.xml -v /env/hadoop/hdfs-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/hdfs-site.xml -v /env/hadoop/mapred-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/mapred-site.xml -v /env/hadoop/yarn-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/yarn-site.xml -v /env/spark/slaves:/opt/spark/conf/slaves -v /env/spark/spark-env.sh:/opt/spark/conf/spark-env.sh -name=master -hostname=master ieds/spark-base:v6 /bin/boot.sh#docker run -idt -v /env/hosts:/etc/hosts -v /env/hadoop/hadoop-env.sh:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/hadoop-env.sh -v /env/hadoop/slaves:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/slaves -v /env/hadoop/core-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/core-site.xml -v /env/hadoop/hdfs-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/hdfs-site.xml -v /env/hadoop/mapred-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/mapred-site.xml -v /env/hadoop/yarn-site.xml:/opt/yarn/hadoop-2.6.0-cdh5.5.0/etc/hadoop/yarn-site.xml -v /env/spark/slaves:/opt/spark/conf/slaves -v /env/spark/spark-env.sh:/opt/spark/conf/spark-env.sh -name=slave1 -hostname=slave1 ieds/spark-base:v6 /bin/boot.sh2. weave设置ip地址#weave connect 31#weave attach /24 master#weave attach /24 slave1 3. 服务启动：进入master主机，并验证各机器间ssh通信无问题1) 格式化master的hdfs#cd /opt/yarn/hadoop-2.6.0-cdh5.5.0/bin#./hdfs namenode format2) 启动hdfs服务#cd /opt/yarn/hadoop-2.6.0-cdh5.5.0/sbin#./start-dfs.sh3) 启动yarn服务同上目录#./start-yarn.sh4) 启动jobserver#./mr-jobhistory-daemon.sh historyserver5) 验证a.命令验证：执行jps命令查看服务是否启动master下： ResourceManager SecondaryNameNode JobHistoryServer NameNodeslave下： DataNode NodeManagerb.Web验证（须部署第六章）：http:/master:80808/cluster http:/master:50070 五、 基于ambari管理平台的镜像搭建1. 部署准备由于ambari部署所需安装包太大，故需要搭建私库下载ambari:/ambari/ubuntu14/2.x/updates//ambari--ubuntu14.tar.gz下载HDP:/HDP/ubuntu14/2.x/updates//HDP--ubuntu14-deb.tar.gz下载HDP-UTILS:/HDP-UTILS-0/repos/ubuntu14/HDP-UTILS-0-ubuntu14.tar.gz2. 私库配置在物理机上安装apache2并配置httpd#vim /etc/apache2/apache2.conf 文件，加入：ServerName localhostDirectoryIndex index.html index.htm index.phpAddDefaultCharset GB2312启动appache2，并将上面下载的文件解压到/var/www/html/目录下3. ambari-server安装1) 修改ambari源#cd /etc/apt/sources.list.d#wget /ambari/ubuntu14/2.x/updates/2.1.2/ambari.list#vim mabari.list修改路径为：32/ambari/ubuntu14（私库地址）#apt-key adv -recv-keys -keyserver B9733A7A07513CAD#apt-get update2) 安装jdk 并配置环境变量 export JAVA_HOME=/opt/jdk1.7.0_79 export CLASSPATH=$JAVA_HOME/lib export PATH=$JAVA_HOME/bin:$PATH3) 配置免登陆，参考上一章4) # apt-get install ambari-server# ambari-server setup5) 配置host和hostname11 ambari#slaves13 hadoop1 14 hadoop215 hadoop36) 制作boot.sh，添加如下内容#!/bin/bashsource /etc/profileservice ssh start /bin/bash7) commit镜像docker commit -m=ambariserver -author=IEDS a790e8142381 ieds/ambari:v18) 验证：#docker run -itd -p 8080:8080 -p 5901:5901 -name= ambari -hostname=ambari ieds/ambari:v2 /bin/boot.sh#weave attach 11/24 ambari#docker attach ambari# ambari-server start访问http:/ ambari:80804. ambari-client安装1) 重复上述1，2，3，5步2) # apt-get install ambari-agent# apt-get install ambari-ntp3) 配置boot.sh#!/bin/bashsource /etc/profileservice ssh startservice ntp startambari-agent start4) commit镜像先执行第六章的操作，然后在执行以下命令docker commit -m=ambariclient -author=IEDS a790e8142381 ieds/ambariclient:v15) 部署实例：#docker run -itd -name=hadoop1 -hostname=hadoop1 ieds/ambariclient:v1 /bin/boot.sh#weave attach 12/24 hadoop15. ambari管理页面访问：32:8080进入管理页面，登陆账号admin/admin