文案分析与火花

Agenda

WhatClouderadoesforSparkEcosystem

AdvancedAnalyticswithSpark

SparkEngineeringinCloudera

ClouderaembracedSparkinearly2014

EngineeringwithInteltobroadenSparkecosystem

Hive-on-Spark

Pig-on-Spark

Spark-over-YARN

SparkStreamingReliability

GeneralSparkOptimization

HiveonSpark

Technology

Hive:“standard”SQLtoolinHadoop

Spark:next-gendistributedprocessingframework

Hive+Spark

Performance

Minimumfeaturegap

Industry

AlotofcustomersheavilyinvestinHive

WanttoleveragetheSparkengine

DesignPrinciples

NoorlimitedimpactonHive’sexistingcodepath

Maximizecodereuse

Minimumfeaturecustomization

Lowfuturemaintenancecost

ClassHierarchy

TaskCompiler

generates

Task

describedby

Work

MapRedCompiler

TezCompiler

MapRedTask

TezTask

MapRedWork

TezWork

SparkCompiler

SparkTask

SparkWork

Work–MetadataforTask

MapReduceWorkcontainsoneMapWorkandapossibleReduceWork

SparkWorkcontainsagraphofMapWorksandReduceWorks

MapWork1

ReduceWork1

MRJob1

MapWork2

ReduceWork2

MRJob2

Query:selectname,

MapWork1

ReduceWork1

ReduceWork2

sum(value)asvfromdec

groupbynameorderbyv;

SparkJob

SparkClientandSparkContext

SparkClient

TalkingtoSparkcluster

Supportlocal,local-cluster,standalone,yarn-cluster,yarn-client

Jobsubmission,monitoring,errorreporting,statistics,metrics,counters

SparkContext

CoreofSparkclient

Heavy-weighted,thread-unsafe

Designedforasingle-userapplication

Doesn’tworkinmulti-sessionenvironment

RemoteSparkContext

BeingcreatedandlivingoutsideHiveServer2

Inyarn-clustermode,Sparkcontextlivesinapplicationmaster(AM)

Otherwise,Sparkcontextlivesinaseparateprocess(otherthanHS2)

AM(RSC)

Node1

AM(RSC)

Node2

Node3

HiveServer2

YARNCluster

Session2

User2

Session1

User1

DataProcessingviaSpark

TreatTableasHadoopRDD(inputRDD)

ApplythefunctionthatwrapsMR’smap-sideprocessing

ShufflemapoutputusingSpark’stransformations(groupByKey,sortByKey,etc)

ApplythefunctionthatwrapsMR’sreduce-sideprocessing

SparkPlan

MapInput–encapsulateatable

MapTran–map-sideprocessing

ShuffleTran–shuffling

ReduceTran–reduce-sideprocessing

Query:Selectname,sum(value)asvfromdecgroupbynameorderbyv;

CurrentStatus

AllfunctionalityinHiveisimplemented

Firstroundofoptimizationiscompleted

Mapjoin,SMB

Splitgenerationandgrouping

CBO,vectorization

Moreoptimizationandbenchmarkingcoming

BetainCDH

/cloudera-labs/hive-on-spark/

/content/cloudera/en/documentation/hive-

spark/latest/PDF/hive-spark-get-started.pdf

AdvancedAnalyticswithSpark

WrittenbyClouderadatascienceteam

FirsteverbookbridgingMLwithHadoopecosystem

Focusingonusecasesandexamplesratherthanamanual

Targetfordatascientistsolvingrealwordanalysisproblems

GenerallyavailableinMay2015

AnalyzingBigData

Buildingamodeltodetectcreditcardfraudusingthousandsoffeaturesandbillionsoftransactions

Intelligentlyrecommendmillionsofproductstomillionsofusers

Estimatefinancialriskthroughsimulationsofportfoliosincludingmillionsofinstruments

Easilymanipulatedatafromthousandsofhumangenomestodetectgeneticassociationswithdisease

ChallengesofDataScience

Datapreprocessing

Variousfastdatafrommultiplesourcerequirespowerfuldatapipeline

Iteration

Fundamentalpartofdatascience

Acceleratingdiskdataloadingismuchhelpful

Fromlabtoproduction

Makedatausefultonon-datascientists

Modelsbecomepartoftheproductionserviceandmayneedtoberebuiltperiodicallyoreveninrealtime.

ValueatRisk

VaR（风险价值或者风险收益）

指在一定的持有期和给定的置信水平下，利率、汇率等市场风险要素发生变化时可能对某项资金头寸、资产组合或机构造成的潜在最大损失。

例如，在持有期为1天、置信水平为99%的情况下，若所计算的风险价值为10万人民币，则表明该银行的资产组合在1天中的损失有99%的可能不会超过10万人民币。

IntroducedbyHarryMarkowitzin1952,NobelPrizeinEconomicsin1990

IllustrationforVaR

MethodsforCalculatingVaR

Variance-Covariance

HistoricalSimulation

MonteCarloSimulation

EstimatingthroughMonteCarloSimulation

instrument

Normalize

marketfactors Input

Modeling MonteCarlo

Simulation

Oil Bonds SNP SNP

Sampling

MonteCarloSimulationwithSpark

Normalizedata

Fillthemissingvalue

Transformthehistoricaldatatotwo-weeks’return

Modeling

Definefactorfeatures

Useregressionmodeltocomputefactorweights

Sampling

Takethecorrelationinformationbetweenthefactorsintoaccount

IfS&Pisdown,theDowislikelytobedownaswell

Simulation

Broadcastinstrumentstoeachnode

Parallelizetrialcomputationacrossworkers

Compu