Scala实战Spark.pdf

实战Scala C spark 1 4 0 bin lib spark assembly 1 4 0 hadoop2 6 0 jar C spark 1 4 0 bin lib datanucleus api jdo 3 2 6 jar C spark 1 4 0 bin lib datanucleus core 3 2 10 jar C spark 1 4 0 bin lib datanucleus rdbms 3 2 9 jar Dscala usejavacp true Xms512m Xmx512m org apache spark deploy SparkSubmit class org apache spark repl Main spark shell 从上面的研究 我们得到spark shell 最终运行了 SparkSubmit 和相应的运行参数 为了让 Spark 运行我们自己的版本 需要修改 Main java 使其把 spark assembly 1 4 0 hadoop2 6 0 jar 指到到我们自己版本的地方 如 C spark 1 4 0 bin lib classes 而SparkSubmit 调用的最重 要的类是SparkILoop scala 位于 C spark 1 4 0 src repl scala 2 10 src main scala org apache spark repl 简单修改 编译后 我们得到了自己的 Spark 运行版本 15 10 20 21 02 07 WARN NativeCodeLoader Unable to load native hadoop library fo r your platform using builtin java classes where applicable Welcome to version 1 4 0 o o Hello I m Tony Spark Using Scala version 2 10 4 Java HotSpot TM 64 Bit Server VM Java 1 8 0 60 Type in expressions to have them evaluated Type help for more information Spark context available as sc 15 10 20 21 02 25 WARN Connection BoneCP specified but not present in CLASSPATH or one of dependencies 15 10 20 21 02 26 WARN Connection BoneCP specified but not present in CLASSPATH or one of dependencies 15 10 20 21 02 46 WARN ObjectStore Version information not found in metastore hive metastore schema verification is not enabled so recording the schema versio n 0 13 1aa SQL context available as sqlContext scala SparkSubmit scala 在目录C spark 1 4 0 src core src main scala org apache spark deploy 下 编译 C spark 1 4 0 src launcher src main java javac classpath C spark 1 4 0 lib classes d C spark 1 4 0 lib classes org apache spark launcher Main java C spark 1 4 0 src repl scala 2 10 src main scala scalac cp c spark 1 4 0 lib classes d c spark 1 4 0 lib classes org apache spark repl SparkILoopInit scala 实战Scala this thread will keep running even after main thread exit var sum 0 for i 1 to 5 sum i i Thr