技术介绍四条白线是内容边界线原则上不要超出

SparkStreaming技术介绍王新义摘要SparkStreaming示例SparkStreaming架构和原理StreamingVSstorm流式处理框架：flume+kafka+streaming示例：socketWordCount示例：JavaKafkaWordCountSparkStreaming体系结构SparkStreamingDataFLowDiscretizedStreams(DStreams)WindowOperationsInputDStreamsFileScoketQueueKafkaflumeTransformationsonDStreamsTransformationMeaningmap(func)ReturnanewDStreambypassingeachelementofthesourceDStreamthroughafunction

func.flatMap(func)Similartomap,buteachinputitemcanbemappedto0ormoreoutputitems.filter(func)ReturnanewDStreambyselectingonlytherecordsofthesourceDStreamonwhich

func

returnstrue.repartition(numPartitions)ChangesthelevelofparallelisminthisDStreambycreatingmoreorfewerpartitions.union(otherStream)ReturnanewDStreamthatcontainstheunionoftheelementsinthesourceDStreamand

otherDStream.count()ReturnanewDStreamofsingle-elementRDDsbycountingthenumberofelementsineachRDDofthesourceDStream.reduce(func)ReturnanewDStreamofsingle-elementRDDsbyaggregatingtheelementsineachRDDofthesourceDStreamusingafunction

func

(whichtakestwoargumentsandreturnsone).Thefunctionshouldbeassociativesothatitcanbecomputedinparallel.countByValue()WhencalledonaDStreamofelementsoftypeK,returnanewDStreamof(K,Long)pairswherethevalueofeachkeyisitsfrequencyineachRDDofthesourceDStream.reduceByKey(func,[numTasks])WhencalledonaDStreamof(K,V)pairs,returnanewDStreamof(K,V)pairswherethevaluesforeachkeyareaggregatedusingthegivenreducefunction.

Note:

Bydefault,thisusesSpark'sdefaultnumberofparalleltasks(2forlocalmode,andinclustermodethenumberisdeterminedbytheconfigpropertyspark.default.parallelism)todothegrouping.Youcanpassanoptional

numTasks

argumenttosetadifferentnumberoftasks.join(otherStream,[numTasks])WhencalledontwoDStreamsof(K,V)and(K,W)pairs,returnanewDStreamof(K,(V,W))pairswithallpairsofelementsforeachkey.cogroup(otherStream,[numTasks])WhencalledonDStreamof(K,V)and(K,W)pairs,returnanewDStreamof(K,Seq[V],Seq[W])tuples.transform(func)ReturnanewDStreambyapplyingaRDD-to-RDDfunctiontoeveryRDDofthesourceDStream.ThiscanbeusedtodoarbitraryRDDoperationsontheDStream.updateStateByKey(func)Returnanew"state"DStreamwherethestateforeachkeyisupdatedbyapplyingthegivenfunctiononthepreviousstateofthekeyandthenewvaluesforthekey.Thiscanbeusedtomaintainarbitrarystatedataforeachkey.Transformations(Window)TransformationMeaningwindow(windowLength,

slideInterval)ReturnanewDStreamwhichiscomputedbasedonwindowedbatchesofthesourceDStream.countByWindow(windowLength,slideInterval)Returnaslidingwindowcountofelementsinthestream.reduceByWindow(func,

windowLength,slideInterval)Returnanewsingle-elementstream,createdbyaggregatingelementsinthestreamoveraslidingintervalusing

func.Thefunctionshouldbeassociativesothatitcanbecomputedcorrectlyinparallel.reduceByKeyAndWindow(func,windowLength,

slideInterval,[numTasks])WhencalledonaDStreamof(K,V)pairs,returnsanewDStreamof(K,V)pairswherethevaluesforeachkeyareaggregatedusingthegivenreducefunction

func

overbatchesinaslidingwindow.

Note:

Bydefault,thisusesSpark'sdefaultnumberofparalleltasks(2forlocalmode,andinclustermodethenumberisdeterminedbytheconfigpropertyspark.default.parallelism)todothegrouping.Youcanpassanoptional

numTasks

argumenttosetadifferentnumberoftasks.reduceByKeyAndWindow(func,

invFunc,windowLength,

slideInterval,[numTasks])Amoreefficientversionoftheabove

reduceByKeyAndWindow()

wherethereducevalueofeachwindowiscalculatedincrementallyusingthereducevaluesofthepreviouswindow.Thisisdonebyreducingthenewdatathatentertheslidingwindow,and"inversereducing"theolddatathatleavethewindow.Anexamplewouldbethatof"adding"and"subtracting"countsofkeysasthewindowslides.However,itisapplicabletoonly"invertiblereducefunctions",thatis,thosereducefunctionswhichhaveacorresponding"inversereduce"function(takenasparameter

invFunc.Likein

reduceByKeyAndWindow,thenumberofreducetasksisconfigurablethroughanoptionalargument.countByValueAndWindow(windowLength,slideInterval,[numTasks])WhencalledonaDStreamof(K,V)pairs,returnsanewDStreamof(K,Long)pairswherethevalueofeachkeyisitsfrequencywithinaslidingwindow.Likein

reduceByKeyAndWindow,thenumberofreducetasksisconfigurablethroughanoptionalargument.OutputOperationsonDStreamsOutputOperationMeaningprint()PrintsfirsttenelementsofeverybatchofdatainaDStreamonthedriver.Thisisusefulfordevelopmentanddebugging.saveAsObjectFiles(prefix,[suffix])SavethisDStream'scontentsasa

SequenceFile

ofserializedobjects.Thefilenameateachbatchintervalisgeneratedbasedon

prefix

and

suffix:

"prefix-TIME_IN_MS[.suffix]".saveAsTextFiles(prefix,[suffix])SavethisDStream'scontentsasatextfiles.Thefilenameateachbatchintervalisgeneratedbasedon

prefix

and

suffix:

"prefix-TIME_IN_MS[.suffix]".saveAsHadoopFiles(prefix,[suffix])SavethisDStream'scontentsasaHadoopfile.Thefilenameateachbatchintervalisgeneratedbasedon

prefix

and

suffix:

"prefix-TIME_IN_MS[.suffix]".foreachRDD(func)Themostgenericoutputoperatorthatappliesafunction,

func,toeachRDDgeneratedfromthestream.ThisfunctionshouldpushthedataineachRDDtoaexternalsystem,likesavingtheRDDtofiles,orwritingitoverthenetworktoadatabase.Notethatthefunction

func

isexecutedatthedriver,andwillusuallyhaveRDDactionsinitthatwillforcethecomputationofthestreamingRDDs.OutputOperations:foreachRDDwordCounts.foreachRDD(newFunction<JavaPairRDD<String,Integer>,Void>(){@Override

publicVoidcall(JavaPairRDD<String,Integer>arg0)throwsException{arg0.saveAsTextFile("/user/root/stream/kafka"+System.currentTimeMillis());//List<Tuple2<String,Integer>>list=arg0.collect();

returnnull;

}

});StreamingVSStorm处理模型,延迟:而Storm处理的是每次传入的一个事件，而SparkStreaming是处理某个时间段窗口内的事件流。因此,Storm处理一个事件可以达到秒内的延迟，而SparkStreaming则有几秒钟的延迟。容错、数据保证在容错数据保证方面的权衡是，SparkStreaming提供了更好的支持容错状态计算。在Storm中,每个单独的记录当它通过系统时必须被跟踪，所以Storm能够至少保证每个记录将被处理一次，但是在从错误中恢复过来时候允许出现重复记录。这意味着可变状态可能不正确地被更新两次。如果你需要秒内的延迟，Storm是一个不错的选择，而且没有数据丢失。如果你需要有状态的计算，而且要完全保证每个事件只被处理一次，SparkStreaming则更好。实现,编程apiStorm初次是由Clojure实现，而SparkStreaming是使用Scala;clojure和scala都是在jvm上执行的，都有javaapi。产品支持Storm已经发布几年了，在Twitter从2011年运行至今，streaming是比较新的项目。集群管理集成尽管两个系统都运行在它们自己的集群上，SparkStreaming能运行在YARN上。单纯的性能比较没有意义

Storm简单介绍参见《Storm的简单介绍.docx》流式处理框架Flume：日志收集系统Kafka：分布式队列系统Sparkstreaming：Storm搭建框架的几点理解高可用性业务系统的融合：input和output处理速度和数据安全（不丢失）消息传递和分发：业务模型兼容性、扩展性、性能、功能综合情况，尽量简化系统Flume Flume是一个分布式、可靠、和高可用的海量日志聚合的系统，支持在系统中定制各类数据发送方，用于收集数据；同时，Flume提供对数据进行简单处理，并写到各种数据接受方（可定制）的能力。FlumeSources：ExecSource、JMS、syslog、AvroSource、HTTPSource、CustomSourceFlumeSink：HDFSSink、LoggerSink、FileRollSink、NullSink、HBaseSink、CustomSinkFlumeChannels：MemoryChannel、FileChannel、CustomChannelFlumeInterceptors：Timestamp、Host、Morphline、RegexFiltering、RegexExtractorKafkaKafka是一个高吞吐量分布式消息系统。kafk