大数据与数据分析

MusingsonDataScienceandStudentsExperiencingDataAnalyticsNewEnglandSENCERCenterforInnovationProf.RandyPaffenrothDataScienceProgramDepartmentofMathematicalSciencesWorcesterPolytechnicInstitutercpaffenroth@2014WPIMyResearchInternetTier3NetworkPSTN/EtfWnMiMi«dtewi|rweii/customer"InternetConnectivityAccesslayer*byUsdriLudovic.ferre•lntemet_Corwieclivity_Overview2_Access.8vg.LicensedunderCreativeCommonsAttribution-ShareAlike3.0vtaWikimediaCommons•httpi/?.rwiki<,File：lnterneLConnectivity_Access_Layer.svg#fnediaviewer/File:lnternet_Conr>ectivityAccesslayer.svgWPIThisisapanel,soIwanttobeprovocative!ProvocativeAdjective1.tendingorservingtoprovoke;inciting,stimulating,irritating,orvexing.So,IwillbealittlesadifIdon’tendupirritatinganyone©WPIThefirstwar:Terminology•Analyzingdatahasalonghistory!•Therehavebeenmanytermsthathavebeenusedtodescribesuchendeavors:•Statistics•ArtificialIntelligence•Machinelearning•Dataanalytics•SinceIhappentoworkina“DataScience”programperhapsImaybeallowedtheindulgenceofusingthatterminology...WPIWhateverwecallit,whatmakesthingsdifferentnow?WPITheG©©dExperiments,observations,andnumericalsimulationsinmanyareasofscienceandbusinessarecurrentlygeneratingterabytesofdata,andinsomecasesareonthevergeofgeneratingpetabytesandbeyond.Analysesoftheinformationcontainedinthesedatasetshavealreadyledtomajorbreakthroughsinfieldsrangingfromgenomicstoastronomyandhigh-energyphysicsandtothedevelopmentofnewinformation-basedindustries.—FrontiersinMassiveDataAnalysis,NationalResearchCounciloftheNationalAcademiesTlh®IB園dlGivenalargemassofdata,wecanbyjudiciousselectionconstructperfectlyplausibleunassailabletheories一allofwhich,someofwhich,ornoneofwhichmayberight.•PaulArnoldSrereWPITheHopefulTheabilitytotakedata一tobeabletounderstandit,toprocessit,toextractvaluefromit,tovisualizeit,tocommunicateit—thatsgoingtobeahugelyimportantskillinthenextdecades,notonlyattheprofessionallevelbutevenattheeducationallevelforelementaryschoolkids,forhighschoolkids,forcollegekids.Becausenowwereallydohaveessentiallyfreeandubiquitousdata.Sothecomplimentaryscarcefactoristheabilitytounderstandthatdataandextractvaluefromit.—HalVarian,Google'sChiefEconomist,/insights/innovation/hal_varian_on_how_the_web_chaltenges_rnanagersMypersonalgoal:Gettingstudentstobeabletothinkcriticallyaboutdata.WPIWhatisBigData?.Thearemanyexamplesof"data",butwhatmakessomeofit“big”?TheclassicdefinitionrevolvesaroundthethreeVs.•Volume,velocity,andvariety.•Volume:Thereisajustalotofitbeinggeneratedallthetime.Thingsgetinterestingand“big”，whenyoucan'tfititallononecomputeranymore.Why?TherearemanyideasheresuchasMapReduce,Hadoop,etc.thatallrevolvearoundbeingabletoprocessdatathatgoesfromTerabytes,toPetabytes,toExabytes.•Velocity:Dataisbeinggeneratedveryquickly.Canyouevenstoreitall?Ifnot,thenwhatdoyougetridofandwhatdoyoukeep?•Variety:Thedatatypesyoumentionalltakedifferentshapes.Whatdoesitmeantostorethemsothatyoucanplaywithorcomparethem?/wiki/Green_Giant#mediaviewer/Plik:Jolly_greengiant.jpgWPIIsBigDatathesameasDataScience?•AreBigDataandDataSciencethesamething?•Iwouldn'tsayso....DataSciencecanbedoneonsmalldatasets.•AndnoteverythingdoneusingBigDatawouldnecessarilybecalledDataScience.IsBigDatathesameasDataScience?•AreBigDataandDataSciencethesamething?•Iwouldn'tsayso...•DataSciencecanbedoneonsmalldatasets.•AndnoteverythingdoneusingBigDatawouldnecessarilybecalledDataScience.•Buttherecertainlyisasubstantialoverlap!WPICanyouevenbecertain?•Forrealworldproblems,Iclaimthatyouwillneverbecertainofanyinferencesfromdata.•Imean,whathappenstoyourcarefullythoughtoutmarketingplanforsomerockingslackswhentheMartiansland.•Whatisunacceptable

iswhenthedatayouactuallyhavedoesnotsupporttheconclusionyoureport.PublicdomainimageWPIItcanbeeasytofoolyourself!Humanbeingsarereallygoodatpatterndetection...Perhapsabittoogood!IsnH,Witten•EiboFrjink•MorkA-HullDATAMININGPracticalMachin*LearningToolsandTechniqueshttp7/A*iki/Cydonia_(region_of_Mars)WPIItcanbeeasytofoolyourself!httpy//wiki/Cydonia_(region_of_Mars)WPISkillsforDataScienceSu