大数据驱动智慧城市

1、城市计算：用大数据驱动智慧城市Big Chalenges in Big CitiesBig Data in CitiesCities OSPeopleThe EnvironmentWinWinWinUrban ComputingTackle the Big challengesin Big cities using Big data!Urban Sensing & Data AcquisitionParticipatory Sensing, Crowd Sensing, Mobile SensingRoad NetworksPOIsAir QualityHuman mobility Traff

2、icMeteorolo Social gyMediaEnergyUrban Data ManagementSpatio-temporal index, streaming, trajectory, and graph data management,.Urban Data AnalyticsData Mining, Machine Learning, VisualizationService ProvidingImprove urban planning, Ease Traffic Congestion, Save Energy, ReduceAir Pollution, .郑宇. 城市计算概

3、述，武汉大学学报. 2015年1月Urban Computing: concepts, methodologies, and applications. Zheng, Y., et al. ACM transactions on Intelligent Systems and Technology.Cities OSPeopleWinWinThe EnvironmentWin UrbanComputingService ProvidingImprove urban planning, Ease Traffic Congestion, Save Energy, ReduceAir Polluti

4、on, .Urban Data AnalyticsData Mining, Machine Learning, VisualizationUrban Data ManagementSpatio-temporal index, streaming, trajectory, and graph data management,.Human TrafficAirMeteorolo SocialEnergyRoadPOIsmobilityQualitygyMediaNetworksUrban Sensing & Data AcquisitionParticipatory Sensing, Crowd

5、Sensing, Mobile SensingZheng, Y., et al. Urban Computing: concepts, methodologies, and applications. ACM transactions on Intelligent Systems and Technology.Manage cross-domain spatio- temporal data usingData: ST propertiesCloud computing platformsIndexing and retrieval algorithmsUrban Big DataData S

6、tructuresSpatio-temporal (ST) PropertiesSpatio-temporal Static DataPOI DistributionsTrajectory DataSpatial-temporal Crowd Souring DataWeather/AQI Station DataRoad Traffic DataPoint-BasedNetwork-BasedFoursquare, Geo-tweets, DianpingSpatio-Temporal Dynamic DataTAXI, DD, Uber,China Mobile, China Teleco

7、mUS EPA, China MEP, IOTSpatial Static Temporal Dynamic DataGaode Maps, Trafficmanagement BureauRoad/Transportation NetworksBing, Google, Gaode,Baidu MapsWhy Urban Big Data PlatformBridge the gap between urban big data and urbancomputing applicationsUrban Big DataUrban Computing ApplicationsMulti-mod

8、al, Large-scale and highly dynamicCity-wide andinstantaneousUrban Big Data PlatformImperative for enabling smart cities!Managing Urban Big Data.DifficultiesLarge-scale and highly dynamic cloud computingCloud computing platforms do not support ST Data wellUnique ST data structures: trajectories (the

9、most complex ST Data)Unique queries: ST-Range queries and KNN queries rather than key wordsData across different domains: Hybrid indexing for managing multi-modality dataTrajectoriesRange QueriesKNN QueriesHybrid indexingSpatial IndexTemporal. IndexJie Bao, et al. Managing Massive Trajectories on th

10、e Cloud, ACM SIGSPATIAL 2016Cloud Computing for ST DataPoint-basedNetwork-based（ POIs）Spatial and Temporal Static（Road Networks）（ Stationary Readings）Spatial Static-Temporal Dynamic（Traffic Readings）（Crowd Sourcing Readings）Spatial-Temporal Dynamic（Trajectories）Azure SQLAzure TableAzure BlobAzure Fi

11、leAzure QueueStorageStorage TypesDataData Source TypesHDInsightVirtual MachinesSpatial IndexSpatio-temporal Index.Spatial IndexTemporal. Index.Value IndexHybrid IndexInterfaceAPILocation-based Range QuerySpatio-temporal Range QueryValue-based Inverted LookupIndexingComputing EnvironmentImperative fo

12、r enabling smart cities!c1c2c3c4Tr1Tr2Tr3Tr4Tr5n1 n2Tr1Tr2 n3Mining the Most Influential k-Location Set fromMassive Trajectories, ACM SIGSPATIAL 2016Jie Bao, Ruiyuan Li, Xiuwen Yi, Yu Zheng.Managing Massive Trajectories on the Cloud.Finding Top-k Most Influential Location SetA submodular maximizatio

13、nproblem, NP-hardInteractive Visual Data Analytics“SmartAdP: Visual Analytics of Large-scale Taxi Trajectories for Selecting Billboard Locations”, VAST 2016Urban Computing for Urban PlanningBest Paper Nominee Award at UbiComp 2011The Most Cited PaperYu Zheng, et al. Urban Computing with Taxicabs, In

14、 Proc. Of UbiComp 2011City-Wide Traffic ModelingPartition a city into regions with major roads Regions are root causes of the problemN. J. Yuan, Yu Zheng Segmentation of Urban Areas Using Road Networks, MSR-TR-2012-65City-Wide Traf ic ModelingProject taxi trajectories onto these regionsBuilding a re

15、gion graph for each time slotFind the skyline points (problematic region pairs)01020304050607001000800600400200-101234|S|E(V) (km/h)E(V)skylineYu Zheng, et al. Urban Computing with Taxicabs, In Proc. Of UbiComp 2011Yu Zheng, et al. Urban Computing with Taxicabs, In Proc. Of UbiComp 2011Cities OSPeop

16、leWinWinThe EnvironmentWin UrbanComputingService ProvidingImprove urban planning, Ease Traffic Congestion, Save Energy, ReduceAir Pollution, .Urban Data AnalyticsData Mining, Machine Learning, VisualizationUrban Data ManagementSpatio-temporal index, streaming, trajectory, and graph data management,.

17、Human TrafficAirMeteorolo SocialEnergyRoadPOIsmobilityQualitygyMediaNetworksUrban Sensing & Data AcquisitionParticipatory Sensing, Crowd Sensing, Mobile SensingZheng, Y., et al. Urban Computing: concepts, methodologies, and applications. ACM transactions on Intelligent Systems and Technology.Texts a

18、nd images spatial and spatio-temporal data;A single data source Data cross different domainsSeparate data mining algorithms machine learning + data managementVisual and interactive data analyticsData AnalyticsData Mining and Machine Learning for Spatial and Spatio-Temporal Data Azure Cloud MLCross-D

19、omain Data fusion MethodsCategoriesCategoriesCategoriesRegionsRegionsFeaturesAX = RUZTime slotsRegionsYY = TRTXfgfptNtNavdvfrwNp Spatial ClassifierRoad Networks: FrPOIs: FpSpatialTemporal ClassifierTraffic: Ft Meteorologic: FmHuman mobility: Fh TemporalAir quality data Traffic DataJoint Task Learner

20、Good, moderation, fast, normal, UnhealthycongestionApplicationsAir Quality: Inference, Prediction and CausalityCity-wide Traffic: Speed, Volume,Cross-Domain Spatio-Temporal Energy and Pollution emissionCorrelation Pattern MiningVisulaizationAzure SQLAzure TableAzure BlobAzure FileAzure QueueStorageS

21、torage TypesHDInsightVirtual MachinesSpatial IndexSpatio-temporal Index.Spatial IndexTemporal. Index.Value IndexHybrid IndexInterface APILocation-based Range QuerySpatio-temporal Range QueryValue-based Inverted LookupIndexingComputing EnvironmentSupport upper-level machine learning modelsExpedite on

22、line feature extractionsPrune unnecessary search spacesMethodologiesforCross-DomainDataFusionStage-based data fusionFeature-level-based data fusionFeature concatenation + regularizationDNN-basedSemantic meaning-based fusionYu Zheng. Methodologies for Cross-Domain Data Fusion: An Overview. IEEE Trans

23、actions on Big DataTemporal ClassifierSpatialTraffic: FtMeteorologic: FmHuman mobility: FhTemporalRoad Networks: FrPOIs:FpSpatial Classifier2kKkNRKkkzr,nmr,nxrMulti-view learning (Co-training)Pro. dependency-based (Topic Models)Air quality dataTraffic DataJoint Task LearnerGood, moderation, Unhealth

24、yfast, normal, congestiontiti+1tjMGg1 g2g1 g2g16g16r1 r2rntiti+1tjr1 r2 rnMrr1r2rnf1 f2fkYXZSimilarity-based (matrix factorization)Transfer Learning-basedM GM rfffrpgConvDenseDenseDenseConvConvConvRanking and Clustering Real Estates using Big DataValues (learned from big data)Increase more in a risi

25、ng marketDecrease less in a falling marketPPrice HouseIncreaseRankH135%R1H529%R1H413%R2.H29%R3H32%R3H6-1.5%R4H7-6.1%R5t0t1t2A) The price of a real estateB) Rank of estates by PRank 1Rank 5Yanjie Fu, Yong Ge, Yu Zheng, et al. Sparse Real Estate Ranking with Online User Reviews and Offline Moving Beha

26、viors. ICDM 2014Geographical UtilityNeighborhood PopularityBusiness Zones ProsperityGeospatial InfrastructureRanking ofReal EstatesClusters of Real EstatesBusiness AreasPublic CommutesTaxi TracesUser Check-ins and CommentsHuman MobilityLocation, Location, LocationPublic CommutesTaxi TracesxUser Chec

27、k-ins and CommentsfeaturesSparsity RegularizationPair-wised ranking constraintih上海市乐虹坊精致生活广场2楼 地铁10号线龙柏新村站2号出口颇辣渝味火锅DNN-Based Urban Flow PredictionImportant for:Traffic managementRisk assessmentPublic safety/Out-flowNew-flowr2End-flowr3r1In-flowPredict In-flow and out-flow of crowds in each region a

28、t next time interval throughout a cityJunbo Zhang, Yu Zheng, et al. DNN-Based Prediction Model for Spatial-Temporal Data. ACM SIGSPAITAL 2016?,O u t Flow石罩,.1坡石复水犀芯又雹，复厚压屯皇过15:30(P)16:0016:3017:0017:3018:0018:3019:0019:3020:0020:3021:0021:3022:0022:3023:0023:30ChallengesUrban crowd flow depends on m

29、any factorsFlows of previous time intervalFlows of nearby regions and distant regionsWeather, traffic control and eventsCapturing spatial propertiesSpatial distance and hierarchyCapturing temporal propertiesTemporal closenessPeriod and trendSpeed (km/h)TimeA) Hourly traffic speed on consecutive days

30、TimeB) Traffic speed at 9-10am on consecutive weekends120.500.80.70.6Ratio48Time Interval(hour)01200.50.60.70.8Ratio4080Distance(km)Data of u1 , u2 , u3 , u4cij : The jth cluster on the ith layerc10High c20Lowc21c22c30 c31 c32 c33c34 c35 c36TrajectoriesConverting Trajectories into Video-like DataJun

31、bo Zhang, Yu Zheng, et al. Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction, AAAI 2017ConvResUnit 1ResUnit Lclose-nessConvResUnit 1ResUnit LperiodConvResUnit 1ResUnit LtrendrecentneardistantJunbo Zhang, Yu Zheng, et al. Deep Spatio-Temporal Residual Networks for Citywide Cr

32、owd Flows Prediction, AAAI 2017ST-ResNet Architecture:A Collective PredictionCapture temporal closeness, period and trendCapture external factorsCapture spatial correlation ofboth near and far distancesKDD 2013/When Urban Air Meets Big DataAir Pollution: A Global Concern !PM2.5, PM10, NO2, SO2, CO,

33、O3Air quality monitor stationS1S2S4S5S3S6S7S8S6S11S13 S12S14S22S15S9S16S16S17S18S19S20 S10S2150kmx40kmWe do not really know the air quality of a location withouta monitoring station!Inferring Real-Time and Fine-Grained air qualitythroughout a city using Big DataMeteorologyTrafficPOIsRoad networksHum

34、an MobilityHistorical air quality dataReal-time air quality reportsZheng, Y., et al. U-Air: when urban air quality inference meets big data. KDD 2013S1S2S6S7S8S6S13 S12 S14S21S19S15S22S9S20S16S10S4S11S3S16S18S5S17Zheng, Y., et al. U-Air: When Urban Air Quality Inference Meets Big Data. KDD 2013Urban

35、 Air SystemSemi-Supervised Learning ModelPhilosophy of the modelStates of air qualityTemporal dependency in a locationGeo-correlation between locationsGeneration of air pollutantsEmission from a locationPropagation among locationsTwo sets of featuresSpatially-relatedTemporally-relateds2s1s3s4ls2s1s3

36、s4ls2s1s3s4tit1t2lTimeGeospaceA location with AQI labels A location to be inferred Temporal dependency Spatial correlationSpatialTemporalTraffic: FtMeteorologic:FmHuman mobility: FhRoad Networks: FrPOIs:FpSpatial ClassifierTemporal ClassifierCo-TrainingZheng, Y., et al. U-Air: When Urban Air Quality

37、 Inference Meets Big Data. KDD 2013EvaluationOverall performance0.00.10.20.30.60.50.40.70.8NO2PM10AccuracyLinearClassicalU-Air DTCRF-ALLGuassian ANN-ALLYu Zheng, et al. In KDD 2013U-Air: when urban air quality inference meets big data.S1S2S4S5S8S5S2S1S7S5S3S3S6S7S6S11S13 S12 S14S22S15S9S16S16S17S18S

38、19S20 S10S21S3S7S6S4S1S8S9S10S1S4S2S6S9S8S1S2S4S3S10S5S9S6S7S8A) BeijingB) ShanghaiC) ShenzhenD) WuhanPublic website: /Forecasting Air Quality Based on Big DataYu Zheng, et al. Forecasting Fine-Grained Air Quality Based on Big Data. KDD 2015Covering 300+ citiesResultsTime1-6h7-12h13-24h25-48hSudden

39、ChangesCitiesBeijing0.750300.62640.5378.30.49681.10.30078.3Tianjin0.746310.63462.10.59567.40.57968.60.43770.9Guangzhou0.805130.74823.90.71426.80.68129.50.47754.6Shenzhen0.8388.40.76417.60.728200.68922.80.57545.3 | | = 1 | =2014-11-14 182014-11-26 162014-12-18 092014-12-29 203002001000400500PM2.5 AQI

40、2014-12-07 08DateTimePrediction Ground TruthA)6-hour PM2.5 prediction of Haidianwanliu Station, BeijingResults Unusual CasesGround truthPrediction6-hour Forecasts during The Military Parade in Sept. 20156-hour Forecasts during The National Holiday (Oct. 1 to Oct. 7)3hResultsMethods1-6 hours7-12 hour

41、sUpdateGrainedHourslevelFFA0.83933.40.79560.01StationWFM0.76149.60.77765.312DistrictBeijing Municipal Environmental Monitoring Center (9/1/2014 to 4/30/2015)Advantages beyond Weather-Forecast-Based Methods (WFM)Finer granularity: station vs district, hour vs half dayFarther predictive capability: 48

42、 vs 24 hoursUpdating frequency: 1 hour vs 24 hoursNeed less data sourcesMore accurateChengdu Environmental Protection Bureau (3/21/2015 to 8/16/2015)Methods1-6 hours7-12 hours13-24 hoursUpdateGrainedHourslevelFFA0.77314.70.67729.50.64531.81StationWFM0.50133.50.52744.50.52343.312DistrictGenerate a pr

43、ediction in 3ms, finishing the forecast of the next 48 hours for a station in 36msCloud + ClientMS Azure小鱼天气Windows PhoneIOSAndroidUniversal Windows Phone App across all platformsDownload the appInferring Gas Consumption and Pollution Emission of Vehicles throughout a CityKDD 2014GoalsEstimate the g

44、as consumption and vehicle emissionson arbitrary road segmentat any time intervalsusing GPS trajectories of a sample of vehiclesGas Consumption3 4 PM2013/09/17Tuesday2013/09/21Saturday2013/10/02National Holiday kg/kmCO3 4 PM2401500kg/km30120Jingbo Shang, Yu Zheng, et al. Inferring Gas Consumption an

45、d Pollution Emission of Vehicles throughout a City. KDD 2014.Gas Consumption & Pollution EstimationJingbo Shang, Yu Zheng, et al. Inferring Gas Consumption and Pollution Emission of Vehicles throughout a City. KDD 2014.A Three-Step ApproachTravel Speed Estimate (TSE) on each road segmentTraffic Volu

46、me Inference (TVI) for each road segmentCalculate the gas consumption and emission of vehiclesSparse TrajectoriesTravel SpeedGas Consumption & Vehicle Emission(1)TSETraffic Volume(2)TVItiti+1tjMGg1 g2 g16 g1 g2 g16r1 r2rntiti+1tjr1 r2 rnMrr1 r2rnf1 f2fkXYZM GM rfffrpgtNtNavdvfrwfgfp NpabcdeCO71.735.411.4-0.2480Hydrocarbon5.573.65-1.1-1.