大数据介绍英文方案课件

BIGDATABIGDATAEVERYMINUTE…1,388

cabs2,777privatecarsDidirideshailed:EVERYMINUTE…1,3882,777DidiriEVERYMINUTE…395,833PeopleloginToWeChat194,444peoplearevideooraudiochattingEVERYMINUTE…395,833194,EVERYMINUTE…625,000YoukuTudouvideosbeingwatchedEVERYMINUTE…625,000EVERYMINUTE…64,814postsandrepostsonWeiboSEARCH4,166,667

searchqueriesEVERYMINUTE…64,814SEARCH4,166EVERYMINUTE…774

peoplebuysomethingonAlibaba’smarketplacesUS$1,133,942spentonAlibabaEVERYMINUTE…774peoplebuyso1Definition2Characteristic3NoSQL4RDBMS5MapReduceCONTENTS6Applications1Definition2Characteristic3NoS1Definition1Definition1DefinitionBIGDATAvolumeofdataimportantdataonaday-to-daybasisforbetterdecisions1DefinitionBIGvolumeofdataim2Characteristic2Characteristic2CharacteristicVolumeThequantityofgeneratedandstoreddata.VarietyThetypeandnatureofthedata.Thequalityofcaptureddatacanvarygreatly,affectingaccurateanalysis.VelocityInthiscontext,thespeedatwhichthedataisgeneratedandprocessedtomeetthedemandsandchallengesthatlieinthepathofgrowthanddevelopment.VariabilityInconsistencyofthedatasetcanhamperprocessestohandleandmanageit.Veracity2CharacteristicVolumeThequant3NoSQL3NoSQL3NoSQLNoSQLreferstodocument-orienteddatabasesSQLdoesn’tscalewellhorizontally.Itisschemaless.Butnotformless(JSONformat).JSON:datainterchangeformatMongoDatabaseCouchDatabase3NoSQLNoSQLreferstodocument3NoSQLBasicAvailabilityspreaddataacrossmanystoragesystemswithahighdegreeofreplication.SoftStateEventualConsistencyBaseModeldataconsistencyisthedeveloper'sproblemandshouldnotbehandledbythedatabase.atsomepointinthefuture,datawillconvergetoaconsistentstate.Noguaranteesaremade“when”.3NoSQLBasicAvailabilityspread3NoSQL{field1:value1,field2:value2…fieldN:valueN}varmydoc={_id:ObjectId("5099803df3f4948bd2f98391"),name:{first:"Alan",last:"Turing"},birth:newDate('Jun23,1912'),death:newDate('Jun07,1954'),contribs:["Turingmachine","Turingtest",…],views:NumberLong(1250000)}JSONStructure3NoSQL{field1:value1,JSONS3NoSQLRDBMSvsNoSQLXszcRowDB:001:10,Smith,Joe,40000;002:12,Jones,Mary,50000;003:11,Johnson,Cathy,44000;004:22,Jones,Bob,55000;index:001:40000;002:50000;003:44000;004:55000;ColumnDB:10:001,12:002,11:003,22:004;Smith:001,Jones:002,Johnson:003,Jones:004;Joe:001,Mary:002,Cathy:003,Bob:004;40000:001,50000…;Smith:001,Jones:002,004,Johnson:003;…3NoSQLRDBMSvsNoSQLXszc3NoSQLBenefitsColumn-orientedorganizationsaremoreefficientwhenanaggregateneedstobecomputedovermanyrowsbutonlyforanotablysmallersubsetofallcolumnsofdata,becausereadingthatsmallersubsetofdatacanbefasterthanreadingalldata.Column-orientedorganizationsaremoreefficientwhennewvaluesofacolumnaresuppliedforallrowsatonce,becausethatcolumndatacanbewrittenefficientlyandreplaceoldcolumndatawithouttouchinganyothercolumnsfortherows.Row-orientedorganizationsaremoreefficientwhenmanycolumnsofasinglerowarerequiredatthesametime,andwhenrow-sizeisrelativelysmall,astheentirerowcanberetrievedwithasinglediskseek.Row-orientedorganizationsaremoreefficientwhenwritinganewrowifallofthecolumndataissuppliedatthesametime,astheentirerowcanbewrittenwithasinglediskseek.3NoSQLBenefitsColumn-oriented3NoSQLSQLvsNonSQLAgoodcompromiseistodesignyoursystemwith3logicalDBs1.NormalSQLDBusedbyyouradminapplicationtocreatecontent.2.No-SQLDBforfront-end/public/high-volumeapplicaitonusedbythepublicinternet.3.ThelastDBisforanalyticalreportingsystemusingcubesandallthatgoodstuff.ThendataflowsfromtheAdminDBtotheclientNo-SQLDBwhensomeone"Publishes"apieceofcontent,theclient(NoSQL)dbprovidesveryfastreadaccessandrecordsuserinteractionswiththecontent.ThenyouhaveascheduledjobthatpullsthedatafromtheclientDBintothereportingsystem.SinceAdmin,client,andreportingareoftenseparateapps,eachapplicationteamcanworkwithdataintheformatthatbestservestheapplicationandthetransitionfromonesystemtotheotherishandledintheservicelayers.3NoSQLSQLvsNonSQLAgoodcom4RDBMS4RDBMS4RDBMSfixed-schema,row-orienteddatabaseswithACIDpropertiesandasophisticatedSQLqueryengineTheemphasisisonstrongconsistency,referentialintegrity,abstractionfromthephysicallayer,andcomplexqueriesthroughtheSQLlanguage.easilycreatesecondaryindexes,performcomplexinnerandouterjoins,count,sum,sort,group,andpageyourdataacrossanumberoftables,rows,andcolumns.4RDBMSfixed-schema,row-orient5MapReduce5MapReduceDividingandconqueringHighlyfaulttolerantEverydatablockreplicatedon3nodesDifficulttoimplement5MapReduceDividingandconqueringHighly5ComparisonRDBMSMapReduceDatasizeGBPBAccessInteractiveandBatchBatchUpdatesRead/WritemanytimesWriteonce,ReadmanytimesStructureStaticSchemaDynamicSchemeIntegratedHigh(ACID)LowScalingNolinerLinerDBARatio1：401：30005ComparisonRDBMSMapReduceData大数据介绍英文方案ppt课件5HowdoesMapReduceworkMapReduceuseskey/valuepairs.(Traditionallyusingrowsandcolumns) --Mapalltheintermediatevaluesforagivenoutputkeyarecombinedtogetherintoalist. --ReduceThereducefunctionthencombinestheintermediatevaluesintooneormorefinalvaluesforthesamekey.--ReduceTwosteps:MapandReduce5HowdoesMapReduceworkMapRed6Application6Application6GovernmentTheuseandadoptionofbigdatawithingovernmentalprocessesisbeneficialandallowsefficienciesintermsofcost,productivity,andinnovation,butdoesnotcomewithoutitsflaws.Dataanalysisoftenrequiresmultiplepartsofgovernment(centralandlocal)toworkincollaborationandcreatenewandinnovativeprocessestodeliverthedesiredoutcome.Belowarethethought[bywhom?]leadingexampleswithinthegovernmentalbigdataspace.6GovernmentTheuseandadoptio6HealthcareBigdataanalyticshashelpedhealthcareimprovebyprovidingpersonalizedmedicineandprescriptiveanalytics,clinicalriskinterventionandpredictiveanalytics,wasteandcarevariabilityreduction,automatedexternalandinternalreportingofpatientdata,standardizedmedicaltermsandpatientregistriesandfragmentedpointsolutions.6HealthcareBigdataanalytics6EducationAMcKinseyGlobalInstitutestudyfoundashortageof1.5millionhighlytraineddataprofessionalsandmanagersandanumberofuniversitiesincludingUniversityofTennesseeandUCBerkeley,havecreatedmastersprogramstomeetthisdemand.Privatebootcampshavealsodevelopedprogramstomeetthatdemand,includingfreeprogramslikeTheDataIncubatororpaidprogramslikeGeneralAssembly.6EducationAMcKinseyGlobalIn6InternetofThingsBigDataandtheIoTworkinconjunction.Fromamediaperspective,dataisthekeyderivativeofdeviceinter-connectivityandallowsaccuratetargeting.TheInternetofThings,withthehelpofbigdata,thereforetransformsthemediaindustry,companiesandevengovernments,ope