《计算机科学导论》课件Unit 16Internet of Things,Cloud Computing and Data Science

?计算机科学导论?课件Unit16InternetofThings,CloudComputingandDataScience216-1Introduction16-2OpportunitiesinIoT,Cloud,andDataScience16-3ChallengesandResearchDirections16-4IoTApplications16-5CloudApplicationServiceModels16-6CloudApplicationDeploymentModels16-7BigDataToolsandTechniques16-8IntegrationofIoT,CloudComputing,andBigData16-9ReferencesandRecommendedReading16-10KeyTerms16-11Summary16-12PracticeSet

OUTLINE5InternetofThingsFigure16.2EnablingtechnologiesofIoT

6CloudcomputingResourcesandservicesareabstractedfromtheunderlyinginfrastructure.Basedonservice-levelagreements(SLAs)andconsumers.Payonlyforwhattheyconsume.Offersvirtualizednetworkresources,ubiquitous(i.e.reliable/efficient/secure)andoptimalresourceutilization,scalability,compatibility,elasticity,on-demandresourcedelivery,resourcesharing,lowercosts,easeofuse,environmentalsustainability,andmanagementautomation7DatascienceThestudyandpracticeofextractingknowledgefromdataandfindingvaluableinsightsindatatohelpmakevaluabledecisions.Wayofextractingvaluableknowledgefrombigdata.Figure16.3Fieldsincorporatedintodatascience8DatascienceDatascientists:Visualizeandinterpretrichdatasets;Managelargebigdatasets;Ensureconsistencyofdatasets;Buildmathematicalmodelsofdatasets.Interdisciplinaryfieldthatincorporatestheoriesandmethodsfrommanyfieldswithinthebroadareasofstatistics,mathematics,informationscience,andcomputerscience.9DatascienceDatascienceandbigdatasometimesusedinterchangeably.Primarydifferenceistheirperspective.Datasciencebeginswiththedatause,whereasbigdatabeginswiththedatacharacteristics.Datascienceisintricatelyintertwinedwithbigdatatechnologies,anddata-drivendecision-making.1016.2 OpportunitiesinIoT,Cloud,andDataScience

OpportunitiesinIoTcreationofanewecosystembyintegratingthevirtualworldwiththephysicalworldcontextawarepartofoureverydayinfrastructure【根底设施】basedonreal-timedata11OpportunitiesinIoTExamplesofIoTsystemsorservices:useofsensorstotrackRFIDtagsplacedonproductssharedvideosurveillanceservice

tomonitorwaterlevelsinnearbyreservoirsRFID-taggedproductsandsmartshelvesequippedwithsensorssuchasmotionandweightdetectorsimplementationofMedicalBodyAreaNetwork(MBAN)tomonitorheartbeat,bloodpressure,temperatureetc.usingCCTVcamerasandfibreopticcablestomonitorreal-timetrafficconditionssmarttransportationsystems12OpportunitiesincloudcomputingSecurityEnergy-efficiency【能效性】IncreasedstorageCostsavingsReliability【可靠性】,fault-tolerance,anddisasterrecovery【灾难恢复】FlexibilityandeaseofuseRemoteaccess【用户接口】andimprovedmobilityAutomaticupdates13OpportunitiesindatascienceandbigdataDatascienceexploreinsightsfrombigdataAllowhiddenprinciplesanddeepcorrelationstobefoundMassiveglobaltransformationtowardsbetterlivingstandardsExamplesofhowbigdataanalyticsischangingthee-commercelandscape:Alibabagroup,AliPay,Alibaba, AliExpress【全球速卖通】,Tmall,Taobao【淘宝】, AliPay,WeChatPayment【微信支付】,and BaiduWallet【百度钱包】1416.3 ChallengesandResearchDirections

ChallengesandresearchdirectionsinIoTSecurityissuesTrustandprivacyIntegrationissuesScalability,datacontrolandsharingDatamanagementDataminingMachine-to-Machine(M2M)communicationsLackofsharedinfrastructureLackofcommonstandardsInteroperabilityMobilityissues15ChallengesandresearchdirectionsincloudcomputingCloudsecurityTransitiontothecloudCloudapplicationsQoS,servicedelivery,andbillingEnergy-efficiencyInteroperability【互操作性】16ChallengesandresearchdirectionsindatascienceDataminingisappliedtoperformactualextractionofknowledgefrombigdataandmachinelearningisappliedtodiscoverpatternsindatasets.Successfuldatascientistsmustbeabletoseeallproblemsintheperspectiveofbigdataanalytics.Adatascientistmustbeabletosystematicallyextractusefulknowledgefromdata.However,ithasbecomedifficultbecauseduetothedatadeluge,dataisnowstoredindifferentdatabases,orInternet.17ChallengesandresearchdirectionsindatascienceNext-generationsemanticdatainfrastructureSmartsearchAlgorithmsScalability,sparsity,andabductivemodelingHeterogeneity,incompleteness【不完全性】orredundancy【冗余】indatasetsSystemarchitecture18ChallengesandresearchdirectionsinbigdataHumancollaborationPrivacyDatadurability【数据持久性】DataavailabilityandstorageFaulttoleranceanddisasterrecoveryTransferissues1916-4IoTApplicationsFigure16.4PotentialIoTApplications2016.5CloudApplicationServiceModelsFigure16.5Clouddeliverytypes【云交付类型】2116.6CloudApplicationDeploymentModelsPrivatecloudServessingleorganization,businessorthelicenseeOrganizationisresponsibleforimplementationMoresecureExpensiveCommunitycloudServesmultipleorganizationswhichhavespecificsharedgoalsorrequirementsCommunitymembersorthird-partyserviceproviderisresponsibleforprovidingrequiredimplementationAffordableandsecure22CloudDeploymentModelsPubliccloudServesgeneralpublicovertheInternetCloudprovidersareresponsibleforprovidingservicesonpay-per-usebasisFlexible,scalable,easytouse,andinexpensivetodeploysecuritymustbewellprotectedHybridcloudCombinationofpublic,private,orcommunitycloudsSupportelasticity,scalability,interoperability,andportability【可移植性】onapplicationanddata2316.7BigDataToolsandTechniquesTheabilitytodesign,develop,andimplementabigdataapplicationisdirectlydependentontheknowledgeoftheunderlyingarchitectureofthecomputingplatform.Bigdataischaracterizedby4Vs:volume【容量】:largeamountofdatavelocity【速度】:speedofdatageneration,processing,transfer,andanalysisvariety【多样性】:heterogeneityindata

veracity【真实性】:levelofaccuracyinthedata24ApacheHadoopAnopen-sourcefundamentalframeworkwritteninJavafordistributedstorageanddistributedprocessingofverybigdatasetsoncomputerclusters【计算机集群】.PrimarycomponentsofApacheHadoop:MapReduceHDFS25ApacheHadoopTable16.1ThefunctionofHadoop’smaincomponentsNameFunctionMapReduceAparallelprocessingsystemoflargedatasets.Itdividescomputationsintotwodistinctsteps;inthefirststep,thelargerproblemisdividedintomanydiscreteindependentpieceswhicharefedtothemapfunctions;thisisfollowedbythereducefunction,joiningthemapresultsbackintoafinalproduct.HDFSAdistributedfilesystemthatprovideshigh-throughputaccesstoapplication.HBaseAscalable,distributeddatabasethatsupportsstructureddatastorageforlargetables.HiveAdatawarehouseinfrastructurethatprovidesadatabasequeryinterfacetoApacheHadoop.MahoutAsuiteofscalablemachinelearninganddatamininglibrary.PigAhigh-leveldata-flowlanguageandexecutionframeworkforparallelcomputation.AvroAdataserializationsystem.ZooKeeperAhigh-performancecoordinationservicefordistributedapplications.AmbariAweb-basedtoolforprovisioning,managing,andmonitoringApacheHadoopclusters.ChukwaAdatacollectionsystemformanaginglargedistributedsystems.StormAdistributedcomputationframework.DataisprocessedinrealtimeinStorm,whileitisbatchedinMapReduce.Additionally,aStormjobrunsindefinitelyuntilkilled,whileaMapReducejobmustendeventually.2616.8IntegrationofIoT,CloudComputing,andBigDataIoT,cloudcomputing,andbigdataareconjoined.Figure16.6Theemerginginfrastructureofsmarterplanet【智慧地球】27IntegrationofIoT,CloudComputing,andBigDataWiththerapidincreaseinthenumberofIoTdevices,theamountofbigdataproducedisalsoincreasing.Storingofsuchmassiveamountsofdataisnotpossiblewithtraditionalstoragemethodsonlocalservers.Cloudisthesolutionforstoring,processingandanalyzingbigdataproducedbyIoTdevicesandapplications.Ratherthanusinglocalstorageattachedtoanelectronicdeviceoracomputer,bigdatausesdistributedstoragetechnologybasedoncloudcomputing.28IntegrationofIoT,CloudComputing,andBigDataCloudprovidesservicesforthecomputation,andprocessingofbigdataproducedbyIoTapplicationsanddevices.Forexample,MapReduce

isusedfortheprocessingofbigdatainacloudenvironment,asitisdesignedfortheprocessingoflargeamountsofdatasetsstoredinparallelinthecluster.However,thereisascarcityoftoolsforbigdataprocessinginclouds.29ChallengesintheIntegrationofIoT,CloudComputing,andBigDataConcernsarisewhensomecriticalandsensitiveIoTapplicationsaremovedtothecloud.Therecanbeseveralreasons,forexample:lackoftrustintheserviceproviderlackofstrongSLAsphysicallocationofthedatabeingunknowntotheuserheterogeneousnatureofbigdatacomingfromdifferentdeviceshavingvariedplatforms,operatingsystems,architectures,andstandards30IntegrationofIoT,CloudComputing,andBigDataItisestimatedthat,inthenextfewyears,therewillbeamassiveincreaseinthenumberofconnectedIoTdevices.IoTwillbeoneofthemainsourcesofbigdata,andcloudwillenabletostoreitforlongtimeandtoperformcomplexanalysesonit.Everyday2.5quintillionbytesofdataarecreated.However,thereisnosingleperfectsolutiontomanagethebigdataoncloudsproducedbyalltheIoTdevicesMoreover,Security,privacyanddataintegrityareverycrucialfactorsforIoTdataonclouds.3116-9ReferencesandRecommendedReadingAcharjyaDP,DehuriSandSanyalS:ComputationalIntelligenceforBigDataAnalysis:FrontierAdvancesandApplications,Adaptation,Learning,andOptimization,Volume19,Springer,2021BessisNandDobreC:BigDataandInternetofThings:ARoadmapforSmartEnvironments,StudiesinComputationalIntelligence,Volume546,Springer,2021BorgiaE:TheInternetofThingsvision:Keyfeatures,applicationsandopenissues.ComputerCommunications,Volume54,1–31,2021ChenM,MaoS,ZhangYandLeungVCM:BigData:RelatedTechnologies,ChallengesandFutureProspects,SpringerBriefsinComputerScience,Springer,2021HassanienAE,AzarAT,SnaselV,KacprzykJandAbawajyJH:BigDatainComplexSystems:ChallengesandOpportunities,StudiesinBigData,Volume9,Springer,202132ReferencesandRecommendedReadingHurwitzJ,NugentA,HalperFandKaufmanM:BigDataForDummies,Hoboken,NJ:JohnWiley&Sons,2021JagadishHV:BigDataandScience:MythsandReality,BigDataResearch,2,49-52LiKC,JiangH,YangLTandCuzzocreaA:BigData:Algorithms,Analytics,andApplications,BigDataSeries,Chapman&Hall/CRC,2021LoshinD:BigDataAnalytics:FromStrategicPlanningtoEnterpriseIntegrationwithTools,Techniques,NoSQL,andGraph,Waltham,MA:Elsevier,2021MachirajuSandGauravS:HardeningAzureApplications,SurenMachirajuandSurajGaurav,202133ReferencesandRecommendedReadingMarinescuDC:CloudComputing:TheoryandPractice,Waltham,USA:Elsevier,2021SaidiAA,FleischerR,MaamarZandRanaOF:IntelligentCloudComputing,FirstInternationalConference,2021SmithIG:TheInternetofThings:2021NewHorizons,Halifax,USA:IERC,2021UnderdahlB:TheInternetofThingsForDummies,Hoboken,NJ:JohnWiley&Sons,2021VermesanOandFriessP:InternetofThings:ConvergingTechnologiesforSmartEnvironmentsandIntegratedEcosystems,RiverPublishers,20213416-10KeytermsApacheHadoopdatascientistMapReducebigdataEnablingtechnologiesofIoTPlatformasaService(PaaS)

cloudcomputingHadoopDistributedFileSystem(HDFS)privatecloudCloudserviceProvides(CSPs)hybridcloudpubliccloudcommunitycloudInfrastructureasaService(IaaS)sensorsCyber-PhysicalSystems(CPS)integrationofIoT,cloud,andbigdatasmartecosystemdatascienceInternetofThings(IoT)SoftwareasaService(SaaS)3516-11SummaryThetransformationfromdigitaltoreal-timeintelligencehasmadeIoT,cloud,anddatascienceindispensabletotheverynatureofwork.Thesetechnologieshaverevolutionizedourglobeandareconsideredascomplementaryastheyhaveenhancedeachothercapacitiesandcapabilities.IoT,sometimescalledInternetofEverything(IoE),representsaparadigminwhichanyphysicalthingcanbecomeacomputerthatisconnectedtoadynamicandself-configuringglobalnetwork.ThingsincludedinIoTaremuchmorethaneverydayobjects,vehicles,electronicequipment,smartphones,mobiledevices,utilitymeters,cameras,Bluetooth,Wi-Fi,Satellite,Ethernet,globalpositioningsystems(GPS),sensors,RFID,wirelesssensornetworks(WSNs),Machine-to-Machine(M2M)communication,2G/3G/4G/5G,IP,industrialsystems,agriculturalsystemsetc.ThetechnologiesthatmakeintelligentIoTapplicationspossibleareconsideredastheenablingtechnologiesofIoT.3616-11SummaryCloudcomputingisamodelinwhichInformationTechnology(IT)resourcesandservicesareabstractedfromtheunderlyinginfrastructureandaccessedondemandbythecustomersbasedonservice-levelagreements(SLAs)establishedthroughnegotiationsbetweenthecloudserviceproviders(CSPs)andconsumers.Thecloudservicecanbehostedon-siteoroff-sitesuchasTencentcloud,Alibaba’sAliyun,Xiaomi’sMiCloud,Apple’siCloud,Microsoft’sSkyDriveandSamsung’sS-Cloud.SomemajorcloudprovidersareAmazonWebServices(AWS),MicrosoftWindowsAzure,Rackspace,andBaiduCloud.Datascienceisthestudyandpracticeofextractingknowledgefromdataandfindingvaluableinsightsindatatohelpmakevaluabledecisions.Itisaninterdisciplinaryfieldthatincorporatestheoriesandmethodsfrommanyfieldswithinthebroadareasofstatistics,mathematics,computerscience,andinformationscience.3716-11SummaryIoTpresentsmanyopportunitiesandbringsaboutnewinnovations.ExamplesofsomeoftheIoTsystemsorservicesare:useofsensorstotrackRFIDtagsplacedonproducts,sharedvideosurveillancese