spark_集群安装与简单测试.docx

Spark集群安装1. 确保服务器已安装jdk若没有安装，请参照相应的文档安装。本次安装使用的jdk版本：jdk1.6.0_312. 安装 Scala 2.11.2Spark 1.2.1 依赖 Scala 2.11.2, 我们必须要安装Scala 2.11.2.上传scala-2.11.2.tgz到服务器，并保存到home目录.1、解压scala压缩包： $ tar -zxf scala-2.11.2.tgz2、root权限将压缩包传到/usr/lib目录下： $ sudo mv scala-2.11.2 /usr/lib3、编辑/etc/profile文件， $ sudo vim /etc/profile 增加系统参数：export SCALA_HOME=/usr/lib/scala-2.11.2export PATH=$PATH:$SCALA_HOME/bin:$JAVA_HOME/binexport JAVA_HOME=/usr/jdk64/jdk1.6.0_31 4、source /etc/profile文件，使环境变量生效； $ source /etc/profile5、测试下scala安装是否成功 $ scala -version 若有如下信息展示，表示scala安装成功：Scala code runner version 2.11.2 - Copyright 2002-2013, LAMP/EPFL3. 安装spark3.1在master上安装并配置Spark上传spark-1.2.1-bin-hadoop2.4.tgz至服务器，解压:$ tar -zxf spark-1.2.1-bin-hadoop2.4.tgz创建一个spark目录，将解压后的目录上传到spark目录中：$ mkdir /home/spark$ mv spark-1.2.1-bin-hadoop2.4 /home/spark设置环境变量:$ vim /home/spark/.bash_profile增加如下参数：export SPARK_HOME=/home/spark/spark-1.2.1-bin-hadoop2.4export CLASS_PATH=.:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$PATH:$SPARK_HOME/bin:$SCALA_HOME/bin:$JAVA_HOME/bin保存后退出#source下.bash_profile。$ source /home/spark/.bash_profile把spark-env.sh.template重命名成spark-env.sh$ cd /home/spark/spark-1.2.1-bin-hadoop2.4/conf$ mv spark-env.sh.template spark-env.sh在spark-env.sh中增加如下参数：$ vim spark-env.sh# 插入如下参数：export SCALA_HOME=/usr/lib/scala-2.11.2 -scala安装目录export JAVA_HOME=/usr/jdk64/jdk1.6.0_31-jdk目录export SPARK_MASTER_IP=28 export SPARK_WORKER_MEMORY=2g export HADOOP_HOME=/usr/lib/Hadoop -指定hadoop的安装目录export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop/ -指定yarn配置文件目录export SPARK_CLASSPATH=$SPARK_CLASSPATH:/usr/lib/hive/lib/mysql-connector-java-5.1.31-bin.jar -指定mysql的jar。为sparkSQL使用其中：SPARK_MASTER_IP为master的ip，SPARK_WORKER_MEMORY为worker节点占用的内存 # 保存退出在/conf/slaves文件中添加Spark worker的hostname, 一行一个。$ vim slavesyudong1yudong2yudong3# 保存退出将hive的配置文件：hive-site.xml复制到spark的conf目录下；Spark的详细配置文件为spark-defaults.conf。可以通过该文件更改spark的默认配置；3.2在所有worker上安装并配置Spark在其他worker节点安装scala和spark，配置参数同上面；注意，三台机器spark所在目录必须一致，因为master会登陆到worker上执行命令，master认为worker的spark路径与自己一样。3.3 启动 Spark 集群在master上执行$ cd /home/spark/spark-1.2.1-bin-hadoop2.4/sbin$ ./start-all.sh打印出如下信息：starting org.apache.spark.deploy.master.Master, logging to /home/spark/spark-1.2.1-bin-hadoop2.4/sbin/./logs/spark-root-org.apache.spark.deploy.master.Master-1-yudong1.outyudong3: starting org.apache.spark.deploy.worker.Worker, logging to /home/spark/spark-1.2.1-bin-hadoop2.4/sbin/./logs/spark-root-org.apache.spark.deploy.worker.Worker-1-yudong3.outyudong2: starting org.apache.spark.deploy.worker.Worker, logging to /home/spark/spark-1.2.1-bin-hadoop2.4/sbin/./logs/spark-root-org.apache.spark.deploy.worker.Worker-1-yudong2.outyudong1: starting org.apache.spark.deploy.worker.Worker, logging to /home/spark/spark-1.2.1-bin-hadoop2.4/sbin/./logs/spark-root-org.apache.spark.deploy.worker.Worker-1-yudong1.out查看进程是否正常：$ jps2735 AmbariServer21060 Jps4766 QuorumPeerMain20784 Master2879 NodeManager20928 Worker5430 NameNode浏览master的web UI(默认http:/localhost:8080). 若8080端口被占用，会自动切换成8081端口，可以在SPARK_HOME/logs下的spark-root-org.apache.spark.deploy.master.Master-1-yudong1.out日志中查看目前使用的端口信息。页面如下所示：从上图可以看到所有的worker节点，以及它们的CPU个数和内存等信息。当有应用在执行时，可以在Running Application下查看到，当应用执行成功后，可以在Completed Application栏查看到。3.4 进入spark-shell工作台 cd $SPARK_HOME/bin执行如下命令：# spark-shell，系统会打印出如下日志：出现上图所示的scala，表示spark集群安装成功。4.spark on yarnSpark在YARN中有yarn-cluster和yarn-client两种运行模式。使用Spark on yarn，要确保环境变量HADOOP_CONF_DIR或YARN_CONF_DIR已经设置；在YARN集群中运行Spark应用程序的命令：SPARK_JAR=./spark-class org.apache.spark.deploy.yarn.Client-jar-class-args-num-workers-master-class-master-memory-worker-memory-worker-cores-name-queue-addJars-files-archives例如：1. SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0.jar2. ./spark-classorg.apache.spark.deploy.yarn.Client3. -jarexamples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar4. -classorg.apache.spark.examples.SparkPi5. -argsyarn-standalone6. -num-workers37. -master-memory2g8. -worker-memory2g9. -worker-cores1在yarn集群上运行spark sql命令：./spark-sql- masteryarn- deploy-mode client- jars- conf- executor-memory-executor-cores- driver-memory -d,-define Variable subsitution to apply to hive commands. e.g. -d A=B or -define A=B -database Specify the database to use -e SQL from command line -f SQL from files -h connecting to Hive Server on remote host -hiveconf Use value for given property -hivevar Variable subsitution to apply to hive commands. e.g. -hivevar A=B -i Initialization SQL file -p connecting to Hive Server on port number -S,-silent Silent mode in interactive shell -v,-verbose Verbose mode (echo executed SQL to the console)例如：spark-sql -master yarn -deploy-mode client -conf spark.shuffle.service.enabled=true -conf spark.dynamicAllocation.enabled=true -executor-memory 1g -executor-