大数据驱动的谷歌云平台运维

大数据驱动的谷歌云平台运维演讲人：通用名DevopsEvolutionDevopsEvolutionDevopsEvolutionCloud／PlatformMicroservicesBig

dataMachine

LearningCloud

and

PlatformsMicroservicesBigDataandMachineLearningDevopsDevOps

(aclippedpoundof

"development"and"operations")

is

asoftwaredevelopment

processthat

emphasizesmunicationandcollaborationbetweenproduct

management,

softwaredevelopment,

andoperations

professionals.---

Wiki“DevOps

is

thebinationof

cultural

philosophies,

practices,

and

tools

thatincreases

anorganization’s

ability

todeliver

applications

andservices

at

highvelocity.”---

AWS“DevOps

is

whereany

application

starts.”---

GoogleGoogle

Cloud

Platformc。mputeStorage。＠。App

Engine

Compute

Eng,neContainer

Engine魄tableCloudStorageCloud

SQLCloudDa旧storeBig

DataMachine

Learning＠＠＠＠＠B唱Query阳阳如bDataflowoacaprocDacalabVi制的APIMad旧neSp幌dTrans坦teLear阳r唱A问APIDataflow

evolutionMapReduceBigTableDremelColossusFlumeMegastoreSpannerPubSubMillwheelApacheBeamGoogleCloudDataflowChoose

your

language...JavaPythonBeamModel创herlanguages...and

your

runtime.I ..IDeployTear

DownFully-managed100

mins. 65

mins.vs.Dynamic

Work

RebalancingUnifiedprogramming

modelCloudLogsGoogleAppEngineGoogleAnalyticsPremiumCloudPub/SubBigQueryStorage(tables)CloudBigtable

(NoSQL)CloudStorage(files)CloudDataflowBigQueryAnalytics(SQL)CaptureStoreAnalyzeBatchCloudMonitoringCloudDataStoreProcessStreamCloudBigtableReal

timeanalyticsandAlertsCloudDataflowCloudDataprocDataflow-IntegratedCloudDataproc16Dataflowpublicstatic

void

main(String[]args)

throws

IOException

{...Pipeline

pipeline

=

Pipeline.create(options);pipeline.apply(PubsubIO.Read.topic(options.getPubsubTopic())).apply(ParDo.of(new

ExtractWords())).apply(ParDo.of(new

Uppercase())).apply(ParDo.of(new

StringToRowConverter())).apply(BigQueryIO.Write.to(tableSpec).withSchema(StringToRowConverter.getSchema()));PipelineResultresult=

pipeline.run();Dataflowjava

-cp./-Dexec.mainClass.example.WordCount

\-Dexec.args="--project=<my-cloud-project>\--stagingLocation=

\--output=

\--runner=BlockingDataflowPipelineRunner"Dataflowstatic

class

ExtractWords

extends

DoFn<String,String>

{@Overridepublic

void

processElement(ProcessContextc)

{String[]words

=

c.element().split("[^a-zA-Z']+");for

(String

word

:words)

{if

(!word.isEmpty())

{c.output(word);}}}}Stackdriverimport.stackdriver.api.custommetrics.CustomMetricsPosterSending

acustom

metricvalue,not

tiedto

aninstance,

collected

nowCustomMetricsPoster(YOUR_API_KEY).sendMetricDataPoint(YOUR_METRIC_NAME,

POINT_VALUE);Sending

multiple

metrics,

all

collected

nowCustomMetricsPoster(YOUR_API_KEY).sendMetricDataPoint(YOUR_METRIC_NAME,POINT_VALUE).sendMetricDataPoint(SECOND_METRIC_NAME,

SECONDPOINTVALUE);}}}}Dataflowstatic

class

ExtractWords

extends

DoFn<String,String>

{@Overridepublic

void

processElement(ProcessContextc)

{String[]words

=

c.element().split("[^a-zA-Z']+");for

(String

word

:words)

{if

(!word.isEmpty())

{c.output(word);CustomMetricsPoster(YOUR_API_KEY).sendMetricDataPoint(“Counting

Word”,1);Dataflowstatic

class

ExtractWords

extends

DoFn<String,String>

{@Overridepublic

void

processElement(ProcessContextc)

{String[]words

=

c.element().split("[^a-zA-Z']+");for

(String

word

:words)

{if

(!word.isEmpty())

{c.output(word);…..Monitoring,CapacityManagement,Rate-limiting,securityenhancementand

etc.}}}}Operationsas-a-ServiceMonitoringMicroServiceAlertingMicroServiceTrafficManagementMicroServiceDe