Spark集群安装方法(详细).doc

本文档是参考娄老师的安装文档和王家林的视频讲解进行整理。建议红色的指令复制粘贴运行，自己敲容易出错，而且也不好检查。我创建的是一个主工作节点sparkmaster，四个从节点sparkwork1，sparkwork2 sparkwork3 sparkwork4，每个节点给了4g的内存，这就需要电脑的内存配置要求比较高。笔记本至少需要8g内存，这时最多也就是三个节点（sparkmaster既是主节点，也是从节点），三个节点的平台搭建参考娄老师的安装文档。也可以根据本文档相应的修改删减sparkwork3和sparkworker4的内容。一、 虚拟化软件、实验虚拟机准备（一） VMware Workstation 11 注册码/key ：1F04Z-6D111-7Z029-AV0Q4-3AEH8l 开发中可使用桌面版VMware Workstation 11，便于向vShpere管理的ESXi Server服务器“上载”PC机中配置好虚拟机，便于把调试好的开发环境迁移到生产环境的服务器上。（二） 模版机安装OS: ubuntukylin-14.04.3-desktop-amd64.iso*在Ubuntu 中安装VMwareTools以便于在宿主机和虚拟机之间共享内存，可以互相拷贝文本和文件，这个功能很方便，具体参见附录1：Linux虚拟机中手动安装或升级 VMware Tools。l 自定义用户lolo及密码ljl，这个安装时候设置，该用户在后面的FTP和SSH服务中用到该用户。（三） 安装VMware tools详见附录一。（四） 安装FTP服务详见附录二二、 Hadoop与Spark的安装配置（一） 登陆和使用系统l 以下用vim和gedit修改相应脚本文件均可，如果是命令行就用vim，如果是图形界面就用gedit。n 进入root用户权限lolololo-virtual-machine:$ sudo sn 安装vim编辑注意：关于校园网linux无法上网的问题，如果你用的是WIFI上网，建议接入360wifi的访问点中。虚拟机使用rootlolo-virtual-machine:# apt-get install vimn 修改lightdm.conf环境变量rootlolo-virtual-machine:# vim /etc/lightdm/lightdm.conf#允许用户登陆并关闭guest用户SeatDefaultsuser-session=ubuntugreeter-session=unity-greetergreeter-show-manual-login=trueallow-guest=falsen 设置root用户密码rootlolo-virtual-machine:# sudo passwd root设置密码：ljln 修改/root/.profile：备注：为避免root登录开机出现以下提示：Error found when loading /root/.profilestdin:is not a ttyrootlolo-virtual-machine:# gedit /root/.profile打开文件后找到“mesg n”，将其更改为“tty -s & mesg n”n 重启rootlolo-virtual-machine:#reboot h now（二） 下载和安装jdk-8u66JDK会被下载到当前用户的Downloads目录下。(可以把所有的集群所需软件的安装包都放在Downloads的目录下，需要某个软件包时，直接拷贝到其安装目录下。以下的软件我们都安装在/usr/local/下的对应目录下，便于查找和统一管理)n 创建java安装目录rootlolo-virtual-machine:# mkdir /usr/local/javan 将压缩包copy到安装目录rootlolo-virtual-machine:# cp /root/Downloads/jdk-8u66-linux-x64.tar.gz /usr/local/javan 进入安装目录rootlolo-virtual-machine:# cd /usr/local/javan 解压缩JDK压缩包rootlolo-virtual-machine:/usr/lib/java# tar -xvf jdk-8u66-linux-x64.tar.gz（也可以用图形化界面来解压缩）n 编辑配置文件，添加环境变量。rootlolo-virtual-machine:# vim /.bashrc“i”加入：export JAVA_HOME=/usr/local/java/jdk1.8.0_66export JRE_HOME=$JAVA_HOME/jreexport CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$JAVA_HOME/bin:$PATH敲“esc” 键 输入“：wq”保存退出。n 使脚本配置生效rootlolo-virtual-machine:# source /.bashrc（三） 配置单机模式hadoop下载链接：/apache/hadoop/common/hadoop-2.6.0/此处下载的hadoop2.6.0已经是64位编译的，可以在64位linux系统下使用。1. 安装SSH和rsyncrootlolo-virtual-machine:# apt-get install ssh或者：sudo apt-get install ssh openssh-server(已经root账户，就可以不用sudo了)（必要时reboot一下，校园网有时更新源有问题）n 启动服务rootlolo-virtual-machine:# /etc/init.d/ssh startn 测试服务rootlolo-virtual-machine:# ps -e |grep sshn 设置免密码登陆rootlolo-virtual-machine:# ssh-keygen -t rsa -P rootlolo-virtual-machine:# cat /.ssh/id_rsa.pub /.ssh/authorized_keysn 测试本地ssh服务：rootlolo-virtual-machine:# ssh localhostrootlolo-virtual-machine:#exitn 安装rsyncrootlolo-virtual-machine:# apt-get install rsync2. 安装hadoop2.6.0注意：目前最新版本为2.7.0，属于测试版本，不稳定，建议使用2.6.0.rootlolo-virtual-machine:# mkdir /usr/local/hadooprootlolo-virtual-machine:# cd /root/Downloads/rootlolo-virtual-machine:# cp /root/Downloads/ hadoop-2.6.0.tar.gz /usr/local/hadoop/rootlolo-virtual-machine:/Downloads# cd /usr/local/hadoop/rootlolo-virtual-machine: /usr/local/hadoop # tar -xzvf hadoop-2.6.0.tar.gzrootlolo-virtual-machine: /usr/local/hadoop # cd /usr/local/hadoop/hadoop-2.6.0/etc/hadoop查JDK路径 rootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#$JAVA_HOMEbash: /usr/lib/java/jdk1.8.0_66: Is a directory3. 编辑Hadoop环境配置文件1）hadoop-env.shrootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#vim hadoop-env.sh备注：此处用gedit命令替代vim也可，看习惯。 键入“i” 将export JAVA_HOME=$JAVA_HOME改为export JAVA_HOME=/usr/local/java/jdk1.8.0_66（其他两个文件加入本句代码）：同时加入export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/lib敲“esc”键，输入“:wq”保存退出。应用该配置：rootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#source hadoop-env.sh2）yarn-env.shrootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#gedit yarn-env.sh在# export JAVA_HOME=/home/y/libexec/jdk1.6.0/下面加入：export JAVA_HOME=/usr/local/java/jdk1.8.0_66rootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#source yarn-env.sh3）mapred-env.shrootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#gedit mapred-env.sh在# export JAVA_HOME=/home/y/libexec/jdk1.6.0/下面加入：export JAVA_HOME=/usr/local/java/jdk1.8.0_66rootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#source mapred-env.sh4）修改/.bashrc文件中的环境变量rootlolo-virtual-machine:/# vim /.bashrcn 插入n export JAVA_HOME=/usr/local/java/jdk1.8.0_66n export JRE_HOME=$JAVA_HOME/jren export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0n export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopn export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativen export HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/libn export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/libn export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATHn 应用配置rootlolo-virtual-machine:# source /.bashrcn 查看Hadoop版本rootlolo-virtual-machine:# hadoop version4. 运行单机例子rootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0#mkdirinputrootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0#cpREADME.txtinputrootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0# bin/hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.6.0-sources.jar org.apache.hadoop.examples.WordCount input outputn 查看结果rootlolo-virtual-machine:/usr/local/hadoop/hadoop-2.6.0# cat output/*至此Hadoop单机模式配置成功*下面不再配置伪分布式hadoop集群，直接配置分布式的集群（四） 配置hadoop分布式集群1. 配置IP地址查看网卡IP配置命令rootlolo-virtual-machine:/# ifconfigeth0Link encap:Ethernet HWaddr 00:0c:29:02:4f:ac inet addr:36 Bcast:55 Mask: 第一种方法：使用管理面板设置IP 打开控制面板，点击“Network” 点击Option，添加IP、网关和DNS 第二种方法：手动设置静态IP（）1） 找到配置文件并作如下修改：rootSparkMaster:/etc/NetworkManager/system-connections# vim Wired connection 1修改如下部分：802-3-ethernetduplex=fullmac-address=00:0C:29:22:2D:C8connectionid=Wired connection 1uuid=de16d53e-bb1a-47c1-a2e8-70b9107b20ectype=802-3-ethernettimestamp=1430738836ipv6method=autoipv4method=manualdns=8;dns-search=8;address1=00/24,本例中使用图形界面修改的，地址配置信息被保在了: /etc/NetworkManager/system-connections/目录下的Wired connection 1文件中。2）重启网卡：sudo /etc/init.d/networking restart2. 修改主机名rootlolo-virtual-machine:/# vim /etc/hostnamelolo-virtual-machine改为：SparkMaster重启后测试：rootlolo-virtual-machine:/#sudo reboot h nowrootSparkMaster:/# hostnameSparkMasterSparkWorker1，SparkWorker2，SparkWorker3，SparkWorker4同上SparkWorker1的IP规划为01SparkWorker2的IP规划为02SparkWorker3的IP规划为03SparkWorker4的IP规划为04（根据自己的IP地址规划，目前是搭建的一台主节点的虚拟机，后面会克隆出其他四个工作节点）rootSparkMaster:/# vim /etc/hosts将： localhost lolo-virtual-machine改为：localhost00 SparkMaster01 SparkWorker102 SparkWorker203 SparkWorker304 SparkWorker43. 安装hadoop2.6.0前面安装单机版的时候已经安装完成了hadoop，此处跳过。4. 编辑Hadoop环境配置文件此处同样跳过。5. 创建分布式文件系统的目录rootSparkMaster:/usr/local/hadoop/hadoop-2.6.0#mkdir tmprootSparkMaster:/usr/local/hadoop/hadoop-2.6.0#mkdir dfsrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0#mkdir dfs/datarootSparkMaster:/usr/local/hadoop/hadoop-2.6.0#mkdir dfs/name或cd /usr/local/hadoop/hadoop-2.6.0 mkdir tmp dfs dfs/name dfs/data 6. 配置分布式部署描述符文件rootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# gedit core-site.xmlrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# gedit hdfs-site.xml rootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# gedit mapred-site.xmlrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# gedit yarn-site.xml 1）core-site.xmlrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0# cd /etc/hadoop rootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# gedit core-site.xml n 分布式 fs.defaultFS hdfs:/SparkMaster:9000 hadoop.tmp.dir /usr/local/hadoop/hadoop-2.6.0/tmp hadoop.native.libtrueShould native hadoop libraries,if present,be used. 2）hdfs-site.xmlrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# vim hdfs-site.xmln 分布式 dfs.replication 3 node.secondary.http-addressSparkMaster:50090The secondary namenode http server address and port. .dir /usr/local/hadoop/hadoop-2.6.0/dfs/name dfs.data.dir /usr/local/hadoop/hadoop-2.6.0/dfs/data node.checkpoint.dirfile:/usr/local/hadoop/hadoop-2.6.0/dfs/namesecondaryDetermines where on the local filesystem the DFSsecondary name node should store the temporary images to merge.If this is acomma-delimited list of directoies then the image is replicated in all of the directories for redundancy.注意：dfs.replication 3把1改为3这样数据就有了3份副本，本例中3）mapred-site.xmlrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop# vim mapred-site.xmln 分布式yarnExecution framework set to Hadoop YARN.mapred.job.trackerSparkMaster:9001Host or IP and port of Job Tracker.4）yarn-site.xmlrootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/etc/hadoop#gedit yarn-site.xmln 分布式 yarn.resourcemanager.hostname SparkMaster yarn.nodemanager.aux-services mapreduce_shuffle 7. 编辑masters和slaves文件sudo gedit /usr/local/hadoop/etc/hadoop/masters分布式：SparkMastersudo gedit /usr/local/hadoop/etc/hadoop/slaves 分布式：SparkWorker1SparkWorker2SparkWorker3SparkWorker4（五） 安装Scala、Spark和IDEA1. 分别解压到相关目录注意：如果想用Scala-2.11.6，需要下载spark-1.3.1源码进行重新编译。解压scala-2.10.4到usr/local/scala/生成usr/local/scala/scala-2.10.4/解压spark-1.6.0-bin-hadoop2.6到/usr/local/spark/生成/usr/local/spark/spark-1.6.0-bin-hadoop2.6/2. 编辑当前用户环境变量配置文件rootSparkMaster:# gedit /.bashrc# for examplesexport JAVA_HOME=/usr/local/java/jdk1.8.0_66export JRE_HOME=$JAVA_HOME/jreexport HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/libexport SCALA_HOME=/usr/local/scala/scala-2.10.4export SPARK_HOME=/usr/local/spark/spark-1.6.0-bin-hadoop2.6export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=.:$SPARK_HOME/bin:$SPARK_HOME/sbin:$SCALA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH使环境变量生效rootSparkMaster:# source /.bashrc3. 编辑spark运行环境配置文件rootSparkMaster:# gedit /usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf/spark-env.shexport JAVA_HOME=/usr/local/java/jdk1.8.0_66export SCALA_HOME=/usr/lib/scala/scala-2.10.4export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.6.0/etc/hadoopexport SPARK_MASTER_IP=SparkMasterexport SPARK_WORKER_MEMORY=4gexport SPARK_EXECUTOR_MEMORY=4gexport SPARK_DRIVER_MEMORY=4gexport SPARK_WORKER_CORES=8rootSparkMaster:/usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf# gedit spark-defaults.confspark.eventLog.enabled truespark.eventLog.dir hdfs:/SparkMaster:9000/historyserverforSparkspark.yarn.historyServer.address SparkMaster:18080spark.history.fs.logDirectory hdfs:/SparkMaster:9000/historyserverforSparkspark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers=one two three#spark.default.parallelism 1004. 编辑Spark的slaves文件gedit /usr/local/spark/spark-1.6.0-bin-hadoop2.6/conf/slavesSparkWorker1SparkWorker2SparkWorker3SparkWorker45. Idea IDEA安装下载路径：/idea/download/安装路径：/usr/local/idea/idea-IC-141.731.2/scala插件下载路径：/files/1347/19130/scala-intellij-bin-1.4.15.zip环境变量配置：gedit /.bashrc# for examplesexport JAVA_HOME=/usr/local/java/jdk1.8.0_66export JRE_HOME=$JAVA_HOME/jreexport HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/nativeexport HADOOP_OPTS=-Djava.library.path=$HADOOP_HOME/libexport SCALA_HOME=/usr/local/scala/scala-2.10.4export SPARK_HOME=/usr/local/spark/spark-1.6.0-bin-hadoop2.6export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=/usr/local/idea/idea-IC-141.1532.4/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$SCALA_HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH备注：这个版本的 .bashrc文件是最完整的！（六） 克隆其他slave节点1. 克隆slave节点如果你使用的VMware虚拟机，可以使用clone的功能，克隆SparkWorker1、SparkWorker2，SparkWorker3，SparkWorker4，建议完全克隆，不是用链接模式，避免依赖。克隆完修改IP和主机名。用Ping命令测试rootSparkMaster:/#ping SparkWorker1ping 主机名Master、SparkWorker1、SparkWorker2、SparkWorker3、SparkWorker4ctl+c结束2. 配置集群SSH无密码验证1）验证备注：参考单机版SSH的配置rootSparkMaster:# ssh SparkWorker1rootSparkWorker1:# exitrootSparkMaster:# cd /root/.sshrootSparkMaster:/.ssh# lsauthorizedauthorized_keys id_rsa id_rsa.pubknown_hosts2）从slave向Master上传公钥文件id_rsa.pubSparkWorker1上传公钥给SparkMaster：rootSparkWorker1: #cd /root/.ssh rootSparkWorker1:/.ssh#lsauthorizedauthorized_keys id_rsa id_rsa.pubknown_hostsrootSparkWorker1:/.ssh#scp id_rsa.pub rootSparkMaster:/root/.ssh/id_rsa.pub.SparkWorker1id_rsa.pub 100% 4070.4KB/s 00:00 SparkWorker2上传公钥给Master：rootSparkWorker2:/.ssh# scpid_rsa.pub rootSparkMaster:/root/.ssh/id_rsa.pub.SparkWorker2id_rsa.pub 100%407 0.4KB/s 00:00SparkWorker3 SparkWorker4同样上传公钥给Master3）Master组合公钥并分发Master上看到公钥已经传过来：rootSparkMaster:/.ssh# lsauthorized id_rsa id_rsa.pub.SparkWorker1 known_hostsauthorized_keys id_rsa.pubid_rsa.pub.SparkWorker2 id_rsa.pub.SparkWorker3 id_rsa.pub.SparkWorker4在Master上综合所有公钥：rootSparkMaster:/.ssh# cat id_rsa.pubauthorized_keys rootSparkMaster:/.ssh# cat id_rsa.pub.SparkWorker1authorized_keys rootSparkMaster:/.ssh# cat id_rsa.pub.SparkWorker2authorized_keysrootSparkMaster:/.ssh# cat id_rsa.pub.SparkWorker3authorized_keysrootSparkMaster:/.ssh# cat id_rsa.pub.SparkWorker4authorized_keysMaster分发公钥给SparkWorker1和SparkWorker2 SparkWorker3 SparkWorker4rootSparkMaster:/.ssh# scp authorized_keys rootSparkWorker1:/root/.ssh/authorized_keysrootSparkMaster:/.ssh# scp authorized_keys rootSparkWorker2:/root/.ssh/authorized_keysrootSparkMaster:/.ssh# scp authorized_keys rootSparkWorker3:/root/.ssh/authorized_keysrootSparkMaster:/.ssh# scp authorized_keys rootSparkWorker4:/root/.ssh/authorized_keys3. 保持配置文件同步如果调试过程中修改了配置文件，需要进行主从同步，需要同步的文件包括：l Hadoop需要的：/.bashrc、hadoop-env.sh、yarn-env.sh、mapred-env.sh、core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、masters、slaves、hosts。l Spark需要的：/.bashrc、spark-env.sh，spark目录下的slaves。更简便的方法是使用root用户拷贝：java、hadoop（scala、spark、idea顺便带上，后面具体介绍）目录到另两台机器上。rootSparkMaster:# scp /.bashrc rootsparkworker1:/root/.bashrcrootSparkMaster:# scp -r /usr/lib/java rootsparkworker1:/usr/lib/rootSparkMaster:# scp -r /usr/local/hadoop rootsparkworker1:/usr/local/rootSparkMaster:# scp -r /usr/lib/scala rootsparkworker1:/usr/lib/rootSparkMaster:# scp -r /usr/local/spark rootsparkworker1:/usr/local/rootSparkMaster:# scp -r /usr/local/idea rootsparkworker1:/usr/local/sparkworker2 sparkworker3 sparkworker4同上三、 Hadoop和Spark集群测试注意：spark1.6.0（spark-1.6.0-bin-hadoop2.6）需要使用scala2.10.x版本。 如果想使用最新的scala2.11.6需要下载spark-1.6.0.tgz，并重新编译，再使用。（一） 启动hadoop分布式集群n 格式化集群文件系统rootSparkMaster:/usr/local/hadoop/hadoop-2.6.0/bin#./hdfs namenode -format或rootSparkMaster:/# hdfs namenode -format15/05/01 18:37:29 INFO namenode.NameNode: STARTUP_MSG: /*STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = SparkMaster/00STARTUP_MSG: args = -format。STARTUP_MSG: version = 2.6.0Re-format filesystem in Storage Directory /usr/local/hadoop/hadoop-2.6.0/dfs/name ? (Y or N) Y15/05/01 18:37:33 INFO namenode.FSImage: Allocated new BlockPoolId: BP-77366057-00-143047665379115/05/01 18:37:33 INFO common.Storage: Storage directory /usr/local/hadoop/hadoop-2.6.0/dfs/name has been succe