大数据助力数字化转型课件

1、数字化企业转型大数据解决方案2变革时代的奏鸣大数据技术简介强大的软件功能坚实的物理基础 新世界的问与答议程Presentation ID跨界颠覆无处不在汽车制造公共交通控制自动化出租车快递航空酒店停车场金融保险能源商场媒体娱乐房产安全部门医疗教育Gartner：2017年十大战略技术趋势数字化转型有很多环节需要演进移动网络数据无处不在生产设备前端应用供应链终端设备安全大数据IoT云移动设备100%of business networks have traffic to malware sites84%believe big data delivers high business value65

2、%of CEOs consider IoT to be strategic57%of organizations will use cloud solutions81%of CEOs believe mobility is strategic行业的快速变更Inclusive Decision-makingAugmented Decision-makingSituational AwarenessBehavioral AwarenessDynamic ProcessesDynamic ResourcesHyper-awarenessInformedDecision-MakingDigitalBu

3、sinessAgilityFastExecution 动态化资源: The ability to acquire, deploy, manage, and re-allocate resources (e.g., talent, technology) as business conditions dictate形势环境感知: The ability to identify changes in an organizations internal and external environments, and to understand which changes matter行为习惯感知: T

4、he ability to understand how workers and customers act, what they think, and what they value动态化流程: The ability to rapidly introduce new business processes and adapt existing business processes to changing business conditions包容化辅助决策: The ability to make decisions based on the shared intelligence that

5、 emerges from the collaboration of disparate individuals and teams数据分析辅助决策: The ability to incorporate data and analytics into the decision-making processes across an organization数字化企业业务模型的三个流程环节数字化之支柱Applications500 billion New generation of applications and servicesAPI Economy Online sales to cros

6、s $500 billion in 2020“Big” Data90%90% of the data created in the last two years. 40 zettabytes by 20207 billion 7 billion people will have access to internet by 2020PeopleThings50 billion devices connected to the internet by 2020.50 billion 大数据应用场景10Media/EntertainmentViewers / advertising effectiv

7、enessCommunicationsLocation-based advertising Education &ResearchExperiment sensor analysisConsumer Packaged GoodsSentiment analysis of whats hot, problemsHealth CarePatient sensors, monitoring, EHRsQuality of careLife SciencesClinical trialsGenomicsHigh Technology / Industrial Mfg.Mfg qualityWarran

8、ty analysis Oil & GasDrilling exploration sensor analysisFinancialServicesRisk & portfolio analysis New productsAutomotiveProduct launchDynamic value chainMonitoring RetailConsumer sentimentOptimized marketingLaw Enforcement & DefenseThreat analysis - social media monitoring, photo analysisTravel &T

9、ransportationSensor analysis for optimal traffic flowsCustomer sentimentUtilitiesSmart Meter analysis for network capacity,On-line Services / Social MediaPeople & career matchingWeb-site optimization算法？建模？我们唯一缺乏的是想象力来自客户的诉求11 传统制造业 设立新的战略数据部 高等教育 开设大数据专业 能源行业 构建智慧运维系统 政府交通 建立智能交通分析系统物联网，工业4.0产生什么？接下

10、来呢？Industry Landscape is Changing各个行业的现状和痛点实体零售收到电商的巨大冲击，亟需转型和谋求更高效的经营手段，大数据分析是方向之一实体零售也在开始做自营电商，现在电商和零售的界限也越来越模糊品牌在大型电商上赚钱的不多，买流量太贵，所以有部分倾向于自己建立渠道全渠道营销是所有零售商都在考虑的事情，甚至早下重金，例如百联集团和家化集团实体零售的数据量偏少，有些甚至为零，如何获得数据是头等大事，特别是外部数据零售和电商都需要商品销售预测、选品规划、营销触发的功能工业4.0刚刚起步，应用模式还在摸索中，核心是CPS(Cyber-physics systems)需要有

11、强大的大数据平台为将来大数据处理的场景做准备需要有好的算法模型做数据挖掘大型电商都有自主的大数据研发团队，靠自己的力量建立大数据平台同零售业类似，电商也需要销售预测和选品规划的功能模块如何有效地对用户进行精准营销如何获得外部数据资源以获得更加精准的用户画像需要更多的流量零售业制造业电商建立在大数据之上的模块市场方面的应用ERP方面的应用工业方面的应用管理模块的切换智能营销客户分析追踪的影响商业洞察全渠道营销进销存管理生产管理商业智能物流管控电子商务订单/商品预测用户需求订单信息产品开发工艺规划生产制造订单/商品预测市场营销领域大数据举例第三阶段：应用第二阶段：数据管理&客户分析第一阶段：数据收

12、集 & 整合实时感知DSPBusiness Intelligence 模块消费者洞察市场细分跨平台分析消费者画像跨媒体跟踪分析Online和offline数据收集传播统一的数据存储管理DMP 平台DMPDMP全渠道营销百联集团上海家化变革时代的奏鸣大数据技术简介强大的软件功能坚实的物理基础 新世界的问与答议程Presentation ID大数据的定义大数据不仅仅是指数据本身，还包括一系列用来收集、管理、挖掘、分析海量信息并解决复杂问题的技术：According to IDC “Big data refers not only to data itself but also to a set o

13、f technologies designed to collect, manage, mine, andanalyze large collections of information to solve complex problems.”IDCAt a recent Big Data and High Performance Computing Summit in Boston hosted by Amazon Web Services (AWS), data scientist John Rauser mentioned a simple definition: 任何大到一台计算机处理不

14、过来的数据就是大数据，Any amount of data thats too big to be handled by one computer. Some says thats too simplistic. Others say its spot on.Amazon Web Services (AWS) “Big data” 是指数据集合的尺寸超过典型数据库软件工具的捕捉、存储、管理和分析能力。refers to datasets whose size is beyond the ability of typical database software tools to capture,

15、 store, manage, and analyze. MGI also says and proves strong evidence that big data can play a significant economic role to the benefit not only of private commerce but also of national economies and their citizens. Data can create significant value for the world economy, enhancing the productivity

16、and competitiveness of companies and the public sector and creating substantial economic surplus for consumers.McKinsey Global InstituteFoundation Research and Analytics Team我们所面对的世界非结构化数据90%，202040 ZBSource:IDC Digital Universe Study我们所面对的世界非结构化数据90%，202040 ZB多结构化数据Variety:文字/图片/视频/文档Petabytes海量信息V

17、olume:传统存储/计算无法处理速度VELOCITY:快速及时有效的分析+ORGANIZE + ANALYZE价值VALUE:单条信息并无太大价值，但庞大的数据量蕴含巨大财富Acquire/AccessProcessDecide大数据的四大特征4个”V” Hadoop是一个分布式存储和分析数据的容错框架。它由两个主要组件构成：Hadoop文件系统（HDFS）数据存储于多个硬件中，其中一个出故障的概率是非常高的。避免数据丢失的常见做法是复制，通过系统保存数据的冗余副本，在故障发生时，可以使用数据的另一份副本。这就是冗余磁盘阵列的工作方式。Hadoop的文件系统HDFS(Hadoop Distr

18、ibuted Filesystem)就是这样工作的。MapReduce应用引擎大部分分析任务需要通过某种方式把数据合并起来，即从一个磁盘读取的数据可能需要和其它多个磁盘中读取的数据合并起来才能使用。MapReduce提供了一个编程模型，其抽象出上述磁盘读写的数据，将其转换为计算一个由成对键值组成的数据集。什么是Hadoop？解决的问题Hadoop的主要组件Hadoop has many building blocksAt the base is a way to Store and Process unstructured dataHadoop Distributed File System(

19、HDFS)At the base is a Self-healing clustered storage system.Map-ReduceDistributed Data Processing PIGHiveSqoopTop level abstractionsTop level InterfacesETL ToolsBI ReportingRDBMSHBASEDatabase with Real-time accessApps API21Flume五大功能Big DataPCPOSSmart PhoneCellularphoneGPSICtagSmart materSensorSNSAV数

20、据源用自动算法代替支持人工决策：复杂的数据分析可以大大优化决策流程、降低风险、挖掘潜在价值新业务模式的创新: 产品、服务客户群分类和精细化服务：通过产品、服务的裁剪，为不同客户群提供更为精细的服务增加透明度：所有应用和客户都可以在第一时间内访问到需要的数据，可以产生巨大价值大数据的价值优势. 大数据的价值优势美国医疗服务业：每年价值3000亿美元大约0.7%的年生产率增长制造业：产品开发、组装成本降低50%运营资本降低7全球个人位置数据：服务提供商收入1000亿美元或最终用户价值达7000亿美元美国零售业：可能的净利润增长水平60%或0.51%的年生产率增长Data Source:McKins

21、ey Global InstituteBig data: The next frontier for innovation, competition, and productivityFoundation Research and Analytics TeamBig Data Financial Services Fidelity National Information Services (FIS)利用大数据监测信用卡欺诈 他们销售基于ParAccel大数据的信用卡风险管理和诈骗监测系统作为信用卡诈骗的新的方法，信用卡运行系统可以根据这一系统实时接受或拒绝信用卡交易According to

22、ParAccel: “With PADB, FIS can engage in two-way conversations with its data to optimize detection for its customers, while minimizing impact on legitimate clients.”大数据在金融服务行业的优势Foundation Research and Analytics Team金融风险管理满足银监会、巴塞尔协议的风险管理要求及时有效的交易风险分析消费行为分析、实时促销和积分管 理交易的实时监控金融产品创新提升客户体验25变革时代的奏鸣大数据技术

23、简介强大的软件功能坚实的物理基础 新世界的问与答议程Presentation ID大数据时代的企业数据库架构EDWOperational(Transactional)ETLEDWBI/ReportsTraditionalNewWebMachineETLBig Data (Hadoop, NoSQL)Operational(Transactional)ETLBI/ReportsDashboardsOperational(Transactional)Operational(Transactional)Transwarp Data Hub架构图最完整的SQL支持99%的SQL 2003支持，唯一支持

24、PL/SQL的引擎（98%），唯一支持ACID分布式事务的SQL引擎；定位数据仓库和数据集市市场，可用于补充或替代Oracle、DB2等分析用数据库。高效内存/SSD计算第一个支持SSD的基于Hadoop的高效计算引擎，可比硬盘快一个数量级；可用于建立各种数据集市，对接多种主流报表工具。最完整的分布式机器学习算法库支持最全（超过50余种）的分布式统计算法和机器学习算法，同时整合超过5000个R语言算法包。适合金融业风险控制、反欺诈、文本分析、精准营销等应用。支持最完整SQL和索引的NoSQL数据库支持SQL2003、索引、全文索引，支持图数据库和图算法，支持非结构化数据存储支持高并发查询最健壮

25、和功能丰富的流处理框架支持真正的Exactly Once语义支持所有组件的高可用(HA)支持流式SQL和流式机器学习Transwarp ProprietaryApache ProjectsTranswarp Manager资源管理 YARN(内置Transwarp Extension)优化存储 HDFS(内置Transwarp Erasure Code)批处理框架MapReduce2协作服务Zookeeper全文搜索Optimized Elastic SearchDiscover数据挖掘机器学习InceptorPL/SQL引擎交互分析、图计算Stream流处理引擎HyperbaseNoSQL数

26、据库综合搜索Guardian安全管控实时同步Data Alive消息队列Kafka日志采集Flume数据集成Sqoop数据集成Data IntegrationSQL开发辅助Waterdrop可视化挖掘Midas交互工具HUE交互分析Zeppelin工作流Oozie内置交互工具Build-in Interactive ToolsAdvantages of TDH1.Complete SQL Support2.Superier Performance3.ACID/Transaction Support4.Distributed Stream SQL5.Rich ML Algorithm Libra

27、ry6. Unified Security Use SQL to create streams, and run ANSI SQL and PL/SQL stored procedures over streaming events.Unified batch and event driven processing on the same engine.HA, low latency, flow controlDistributed TransactionsBatch/Incremental CRUD OperationsMVCC & Two Phase Locking to guarante

28、e consistencyBest performance for 1TB/10TB/100TB TPC-DS benchmark tests。In-memory/On-SSD columnar store for low latency interactive analysisANSI SQL 2003（99%）Oracle PL/SQL（98%）DB2 SQL/PL（90%）Teradata SQL（90%）Easy to migrate legacy applications or develop new appsSQL to setup security rules for all c

29、omponents.Single Sign OnRole based Access ControlRow/column granularity access control.70 distributed statistics and machine learning algorithmsSeamless R integration to call distributed algorithms. Connectors for Rapdminers for ML pipeline creation/modeling.Fusion Distributed Execution Engine 分布式执行

30、引擎Association Mining关联/推荐Classification分类算法Clustering聚类算法Sequential Analysis时序分析Regression回归算法Deep Learning深度机器学习DimensionReduction主成分分析Statistics统计算法R Runtime Library R语言动态运行库Belief Network信念网络Graph 图计算Sampling采样算法Discriminate Analysis判别分析Reinforcement 增强学习Decision Methods决策方法Factor Analysis因子分析Gen

31、etic 遗传算法Java/Scala Interfaces Rapidminer Graphical IDERstudio IDEHubble Core算法计算接口Graph engine图计算引擎Customized Plugins自定义插件Transwarp Connector SQL Interfaces to connect data sourcesIndustry Templates行业模板Feature Eng特征工程StreamInceptorHyperbaseSQL Interface Transwarp Discover Toolkits精准营销欺诈检测文本挖掘实时推荐信用

32、风险流失预警客户精分异常行为识别智能维护系统Transwarp Discover 机器学习工具优势一：完整的数据库支持能力，包括SQL2003、PL/SQL支持和超强的性能优势二：在Hadoop上保持数据一致性优势三：交互式数据分析和挖掘能力与数据可视化工具良好对接在数据可视化的过程中Spark扩展支持大量的可视化及报表生成工具，如 Tableau，SAP Business Objects, Oracle Business Intelligence等，使得基于大数据分析的商业决策更易被理解和接受，从而将大数据的潜在价值最大化。业务人员通过简单的拖拽既可定制个性化报表，跳过了数据准备的工作环节。

33、优势四：完整的数据挖掘和机器学习算法Make Machine Learning More AccessibleR Runtime Library R语言动态运行库Belief Network信念网络Decision Methods决策方法Sampling采样算法Discriminate Analysis判别分析Q-Learning增强学习Graph Inference图推理Factor Analysis因子分析Genetic Algorithm遗传算法Transwarp Hadoop 分布式系统Transwarp DiscoverDistributed Algorithm LibraryAss

34、ociation Mining关联/推荐Classification分类算法Clustering聚类算法Sequential Analysis时序分析Regression回归算法Deep Learning深度机器学习DimensionReduction主成分分析Statistics统计算法Data EngineersData ScientistsWorkflow Tools to build pipelinesTranswarp InceptorSQL EngineData FrameAbstractionData Transformation using PL/SQLFeature Extr

35、action using data frame and native R operationsMachine Learning using more distributed algorithmsData Mining using native R algorithms民生银行持卡人行为分析训练数据采样民生银行2012年的0409半年的交易流水，一共大约2亿条记录，506万个独立持卡人，数据大小约80G。并行360度用户画像在2分钟内完成对506万独立持卡人的画像消费频繁度消费水平美食爱好旅游爱好体育爱好电子爱好IT爱好年轻活力男性女性商人开车一族电话达人差旅人士优势五：高并发低延时的NewSQ

36、L分布式数据库Hyperdrive Project for HyperbaseTranswarp HyperdriveIndexable Storage Engine implemented for Hyperbase HBaseElastic SearchTransactionSQL & APITransactionExecution Engine分布式事务处理引擎IndexSQL & APIGlobal/LocalIndex全局/局部索引SearchSQL & APIDistributedFull-text Search全文搜索InceptorStarGateProjectHyperbas

37、e Native类型支持全面兼容全文索引，支持正则表达式作为语法全面提升模块易用性民生银行卡部历史工单查询历史数据量（4年）数据表行数大小dds_acct_acct181246212.7Gdds_acct_card3866529917.6Gdds_acct_stm8Gdds_trans_event716425258218.5Gdds_acct_quick_chng1070666344616.5Gtab_info_list3094239.4K合计925GB生产系统(SAS)TDH硬件2x P750小型机（HA)8台x86服务器工单查询延时最快20分钟平均4秒程序SAS

38、444行PL/SQL 108行民生银行理财业务数据并发查询单位：SQL查询/秒DPF集群使用power 7+处理器，共64个物理核；TDH集群使用x86 E5处理器，共72个物理核，CPU性能DPF集群比TDH集群强3倍左右优势六：支持SQL和R的实时数据处理技术优势七：图形化运维、严格的安全管控运维和高级存储技术图形化管理、监控、运维工具数据仓库Inceptor分布式文件系统HDFS实时在线查询库Hyperbase流处理Stream企业活动目录LDAP用户认证Kerberos用户A用户B用户N统一资源调度和管理YARN我们的核心优势：全面的多租户安全Inceptor支持基于 Kerberos

39、/LDAP 强身份认证机制支持Spark内部数据加密传输更加完善的基于角色的访问控制支持设置超级用户支持SQL级别的权限控制支持创建、删除、授予、取消角色支持授权角色权限控制StreamKafka支持 Kerberos/基于IP的身份认证机制支持对消息队列的权限控制 (读、写、创建、删除）支持与zookeeper的通信进行kerboros认证更加完善的身份认证和权限管理APIHyperbase支持单元格级别的访问控制服务端透明加密HDFS 支持对HDFS文件设置访问控制列表 HDFS 和 YARN 间的通信交互支持Kerberos认证Yarn作业提交权限控制队列使用权限控制45变革时代的奏鸣大

40、数据技术简介强大的软件功能坚实的物理基础 新世界的问与答议程Presentation ID大数据平台: 什么最重要 ?As big data solutions become mainstream predictable performance will become a table stakeCores, IO and NW BW bandwidth, IOPS性能Storage capacity demands will grow quickly, so capacity planning for today and future growth is criticalRaw capacit

41、y, RAID, replication, retention容量Deployments will grow to hundreds to thousands of servers and terabytes to petabytes the fabric must be able to support the scalingNetwork bandwidth and scaling扩展Reducing TCO is important. Lower CapEx by using optimal infrastructure and software platforms, Lower OpEx

42、 by automation$/performance, capacityTCOIT will need to quickly, and cost-effectively scale resources as business users demandAutomation, management and monitoring管理Forrester Wave: Big Data Hadoop-Optimized SystemsCisco UCS in the leader category“Cisco Systems provides a viable midsized system at at

43、tractive price point. Cisco UCS Integrated Infrastructure for Big Data provides a secure and scalable infrastructure to support enterprise requirements. Ciscos UCS solution comes pretested and prevalidated for Cloudera, Hortonworks, IBM, and MapR, providing a lower-cost and scalable storage platform

44、 to support Hadoop deployments. Management tools such as Cisco UCS Manager and Cisco UCS Director allow for simple configuration of big data Hadoop clusters that can adapt dynamically to changing workloads. Ciscos key differentiators lie in its ability to offer a wide range of configurations, its st

45、rong focus on internet-of-things (IoT) use cases, and its broad partner ecosystem.”SingleConnect: LAN, SAN and ManagementUCS 6200 and 6300 Series Fabric Internments,Installed in pairs, active-active.UCS Manager is embedded Support for direct connectivity to Fabric Interconnects or through Nexus 2000

46、 Series Fabric ExtendersPre-tested and pre-validated configurationFabric-based infrastructure integrates computing, networking, and storage resourcesDesigned for high performance and availabilityCisco UCS Integrated Infrastructure for Big DataTopology ProvisioningMonitoringMaintenanceGrowth49Abstrac

47、tion identities of the application server into a service profile that speeds deployment, reduces errors, lowers costsCisco UCS Advantage - Unified ManagementRobust management delivers superior programmability, scalability, and automation for Big Data deployments TraditionalUCS Manager Service Profil

48、e LANSANNIC MACsHBA WWNsServer UUIDVLAN AssignmentsVLAN TaggingFC Fabrics AssignmentsFC Boot ParametersNumber of vNICsBoot orderPXE settingsIPMI SettingsNumber of vHBAsQoSCall HomeTemplate AssociationOrg & Sub Org Assoc.Server Pool AssociationStatistic ThresholdsBIOS scrub actionsDisk scrub actionsB

49、IOS firmwareAdapter firmwareBMC firmwareRAID settingsAdvanced NIC settingsSerial over LAN settingsBIOS SettingsServer UUIDSerial over LAN settingsBoot orderIPMI settingsBIOS scrub actionsBIOS firmwareBIOS SettingsRemoteKVM IP settingsCall Home behaviorRemote KVM firmwareNumber of vNICsPXE settingsNI

50、C firmwareAdvanced feature settingsVLAN assignments for NICsVLAN tagging config for NICsNumber of vNICsPXE settingsNIC firmwareAdvanced feature settingsFC Fabric assignments for HBAsNumber of vHBAsHBA WWN assignmentsFC Boot ParametersHBA firmwareRAID settingsDisk scrub actionsLANSANQoS settingsBorde

51、r port assignment per vNICNIC Transmit/Receive Rate LimitingHSph-Hadoop Sort per HourThis provides a normalized value of how much data is generated, sorted, and validated in one hour for the scale factor (divide by 30 for a 30TB run). This is the performance of the system under test. Higher HSph is

52、betterPrice/HSph: Price per Performance This divides the total cost of the System under test (inclusive of hardware, software, license cost, and 3 year 24x7x4 support) along with discount and divide by Performance (above HSpH). Lower price/performance is better*As of 10-Juy-2016. Visit for latest re

53、sults TPC Express Benchmark HS: Industrys first standard for benchmarking big data systemsto provide the industry with verifiable performance, price-performance and availability metrics of hardware and software systems dealing with Big Data业界标准指标测试结果 TPC Members高性能Optimized for fast query execution

54、and unmatched data loading弹性扩展在统一管理平台下可以支持高达上万个节点高可用性无论管理平面还是数据平面都采用全冗余的架构设计统一网络：Unified Networking数据、管理、KVM、Image快速部署通过独特的Service Profile技术实现快速部署统一管理：Unified Management计算、网络和I/O的统一管理系统安装和微码分发广泛的合作伙伴Oracle、EMC、MAPR、ParAccel、ClouderaUCS For Big Data的优势TDH on Cisco UCS practice*Other names and brands

55、may be claimed as the property of others.TDH on Cisco UCS practice54*Other names and brands may be claimed as the property of others.*其测试指标采用TPC-DS评判标准，其提升幅度相对于原有的系统产品和性能信息1在性能检测过程中涉及的软件及工作负载可能只对CISCO UCS C240 M4的性能进行了优化。性能测试使用特定的计算机系统、组件、软件、操作系统和功能进行测量。对这些因素的任何更改可能导致不同的结果。如欲了解更多信息，请访问www.transwarp.

56、io 。2星环信息科技不对TPC-DS性能指标评测或网站的设计或实施工作承担任何管理或审核责任。3.此产品中依赖于处理器和平台的优化仅适用于cisco UCS C240 M4平台。4 TPC* 基准测试名称TPC-DS*是TPC标准性能评估机构的注册商标。Better performance, Better choice星环TDH 4.6v大数据平台在cisco C240 M4平台性能再创佳绩，性能有2.6倍提升*星环TDH 4.6v首次4个节点的思科服务器上实现11小时内完成10TB数据的性能测试星环TDH在cisco UCS 总容量80TB数据空间下完成了60TB数据搜索星环TDH平台全面

57、支持cisco FI service profile和无状态计算功能55*Other names and brands may be claimed as the property of others.*其测试指标采用TPC-DS评判标准，其提升幅度相对于原有的系统产品和性能信息1在性能检测过程中涉及的软件及工作负载可能只对CISCO UCS C240 M4的性能进行了优化。性能测试使用特定的计算机系统、组件、软件、操作系统和功能进行测量。对这些因素的任何更改可能导致不同的结果。如欲了解更多信息，请访问www.transwarp.io 。2星环信息科技不对TPC-DS性能指标评测或网站的设计或实施工作承担任何管理或审核责任。3.此产品中依赖于处理器和平台的优化仅适用于cisco UCS C