3-6-基于 ElasticStack 的数据探索与分析-曾勇

11大数据平台架构大数据平台架构22016-4-2222016-4-223Aboutme勇(Medcl)•Developer@Elastic‒JoinedElasticsinceSeptember,2015•medcl@elastic.co•BasedinChangsha,Hunan,ChinaWhat’sElastic？•Adistributedstartupcompany，since2012‒Withemployeesin27countries(andcounting),spreadacross18timezones,speakingerlanguages•WeareworkingonOpenSourceprojects!‒(Luckilysomeofthemarepopular,eg:elasticsearch)•OfferingsupportSubscription，X-pack，CloudandTrainings•Finduson:/elasticandhttps://www.elastic.co45ButELKisout!Beats&PacketbeatELKB?BELK?LKBE?BKEL?678ReleaseBonanza99It’stimetounite!9The“ElasticStack”，staytogetherfromv5.0ElasticStackGithub:EnablePowerfulSearchForBothEnd-UsersAndDevelopershttps://www.elastic.co/use-cases/githubNASA:UnlockingInterplanetaryDatasetswithReal-TimeSearchPic：/msl/multimedia/images/?ImageID=7693https://www.elastic.co/elasticon/2015/sf/unlocking-interplanetary-datasets-with-real-time-search13Datadog：analysismetricsandtime-seriesdataThe“ElasticStack”77•Collectsdiversesources•Collectsdiversesources–Logs+manyothers–Over200plugins•Connectswithlivestreamsdata–Wire/Transactiondata–Full-PacketNetworkCapture/elastic/logstashLogstash:CollectfromdiverseinputstaesensorsThe“ElasticStack”•BeatsarelightweightshippersthatcollectandshipallkindsofoperationaldatatoElasticsearch‒Smallapplication‒Installasagentonyourservers‒WritteninGolang‒Noruntimedependencies‒Singlepurpose/elastic/beatsExamplesofoperationaldata20Packetbeat：Real-timeapplicationmonitoringSniffsthetrafficbetweenyourservers,parsestheapplication-levelprotocolsonthefly.Built-inprotocols：•HTTP•MySQLLet’sgorealtime!•PostgreSQL•Redis•Thrift-RPC•MongoDB•DNS•Memcache•ICMP•AMQP21•…winlogbeat!22ForwardsWindowsEventlogstoElasticsearch23FilebeatAmorelightweightlogshipper•GenericfilteringFlexiblyreducetheamountofdatasentofthewireandstoredTopbeatperiodicallytoElasticsearch.AlsoworksonWindows.SystemwidesystemloadtotalCPUusage…processstatenamecommandline…Diskusageavailabledisksused,freespace…24That’sMore!Metricbeat:ConnectingNumb3rs•Listenstotheinternal“beat”ofsystemsviaAPIs./elastic/beat-generator/2526The“ElasticStack”What’sKibana？icsonplatformithElasticsearch./elastic/kibana/elastic/generator-kibana-plugin27Search&Exploration28Visualization&Dashboard2930The“ElasticStack”Elasticsearchisanopensource,distributed,scalable,highlyavailable,document-oriented,RESTful,fulltextsearchenginewithreal-timesearchandanalyticscapabilitiesThomsonReuters:“107clusters~1747nodes”@Elastic{ON}16lasticthomsonreutersresearchjournalismfinanceandelasticNetflix：”~150clusterstotaling~3,500nodeshosting~1.3PBofdata”/2016/02/evolution-of-netflix-data-pipeline.html?m=131/elastic/elasticsearch•Real-timeanalytics•Timeseriesdataanalytics•Logginganalytics•Securityanalytics•Frauddetection•Predictionmodeling•Recommendations•…332慢慢着，Elasticsearch不是搜索引擎么？3333YouYouknowforsearch,andanalytics！v1.0.0：Aggregationv2.0.0:PipelineAggregation33Aggregation任何数据能被查询到的数据就能被分析接近实时按需实时计算，~1s刷新间隔可嵌套组合不像facets只有一级343535AggregationBuckets：min-min-avg-maxStatsCardinalityHistogramGeohashgrids……SELECTCOUNT(*),AVG(score)<---MetricsFROM`table`GROUPBYprovince,city<---Buckets353636Aggregation==3万英尺高空俯视==PatternsFindsomebeauty(insights)！363737 Orlikethis!373838“aq_level”:“严重污染”,“aq_rank”:68,“aqi”:391,“co”:115.5,“no2”:1.888,“o3”:62.2,“pm2_5”:415.7,“range”:“74~500”,“so2”:523.5,38数据来源：3939StructureofanAggregation39空气质量统计(北京全年)POSTdemo/_search?size=0{"aggs":{"aq_stats":{"terms":{"field":"aq_level",}}}}404141平均空气质量统计(按城市)(nested){{"aggs":{"city_stats":{"terms":{"field":"city",},"aggs":{"avg_pm25":{"avg":{"field":"pm2_5"}30天空气质量趋势分析(Pipeline){"aggs":{"qa_date_histo":{“date_histogram”:{"field":"date“,"interval":"day“},"the_avg":{"field":"pm2_5“"the_movavg":{"moving_avg":{"buckets_path":"the_avg","window":304242Aggregation工作原理•LuceneCollector•Optimizeddatastructure–Compressedcolumnardatastore(previousFieldData，nowDocValues)persegment•Singlepassonyourdata，alonewiththequery4344ShardShardAggregation工作原理ShardShardCCoordinatorShardShard45ShardShardAggregation工作原理ShardShardCCoordinatorShardShard46ShardShardAggregation工作原理ShardShardCCoordinatorShardShard47ShardShardAggregation工作原理ShardShardCCoordinatorShardShard48SegmentSeSegmentSegmentAggregation工作原理Aggregation：s1234AAggregationCollectorTopHitsCollectorSSegmentLuceneIndex/AnESShardSearch：InvertIndexrm2，4，6AggregationAggregation工作原理POSTdemo/_search?size=0{"aggs":{"city_stats":{"terms":{"field":"city","size":10},"aggs":{"avg_pm25":{"avg":{"field":"pm2_5"}},"max_pm25":{"max":{"field":"pm2_5"}}}}}}49avg(pm2_5)terms(city)max (pm2_5)AggregationAggregation工作原理POSTdemo/_search?size=0{"aggs":{"city_stats":{"terms":{"field":"city","size":10},"aggs":{"avg_pm25":{"avg":{"field":"pm2_5"}},"max_pm25":{"max":{"field":"pm2_5"}}}}}}50avg(pm2_5)terms(city)max (pm2_5)AggregationAggregation工作原理POSTdemo/_search?size=0{"aggs":{"city_stats":{"terms":{"field":"city","size":10},"aggs":{"avg_pm25":{"avg":{"field":"pm2_5"}},"max_pm25":{"max":{"field":"pm2_5"}}}}}}51avg(pm2_5)terms(city)max (pm2_5)AggregationAggregation工作原理POSTdemo/_search?size=0{"aggs":{"city_stats":{"terms":{"field":"city","size":10},"aggs":{"avg_pm25":{"avg":{"field":"pm2_5"}},"max_pm25":{"max":{"field":"pm2_5"}}}}}}52avg(pm2_5)terms(city)max (pm2_5)What’smore?‒唯一值：Cardinality‒Hyperloglog++‒百分位：Percentilememory‒Terms‒Sampler‒Maxdocspershard‒SignificantTermsAggregationrid535545555555657585960616263The“ElasticStack”64Community•源码&Issue:/elastic/•英文社区:http://discuss.elastic.co•中文社区:•下载:https://www.elastic.co/downloads•博客:https://www.elastic.co/blog•线下活动:/•IRC:#elasticsearch,#logstash,#kibana,#beats65656666Thanks!6667676768POSTPOSTdemo/pm2_5/1{“city”:“北京”,“date”:“2016-02-08”,“aq_level”:“严重污染”,“aq_rank”:68,“aqi”:391,“co”:115.5,“no2”:1.888,“o3”:62.2,“pm2_5”:415.7,“range”:“74~500”,“so2”:523.5,“location”:{“lat”:39.92,“lon”:116.46}}ES基本操作索引(插入数据){{"_index":"demo","_type":"pm2_5","_id":"1