下一代网格计算与云计算

1、下一代网格计算与云计算人类对计算机资源的不变追求？PowerfulScalableAccessibleReliable23提纲云计算概述云计算体系结构云计算关键技术 - 虚拟化云计算关键技术 - MapReduce云计算在北理Gartner ReportTop 10 Strategic Technology Areas for 2009 VirtualizationCloud ComputingServers: Beyond BladesWeb-Oriented ArchitecturesEnterprise MashupsSpecialized SystemsSocial Software

2、and Social NetworkingUnified CommunicationsBusiness IntelligenceGreen Information TechnologyTop 10 Strategic Technology Areas for 2010Cloud Computing Advanced AnalyticsClient Computing IT for GreenReshaping the Data CenterSocial ComputingSecurity Activity Monitoring Flash MemoryVirtualization for Av

3、ailabilityMobile ApplicationsGartner Report纽约时报租用亚马逊的云计算服务，使用基于云计算的开源软件Hadoop，将其自1851年以来的1100万份报道转变成可搜索的数字化文档，耗时仅一天。如果用传统方法，这项工作可能要数月才能完成。 最近兴起的云计算甚至可以让你体验每秒10万亿次的运算能力，拥有这么强大的计算能力可以模拟核爆炸、预测气候变化和市场发展趋势。云计算4People云作为数据中心（多终端同步-透明）PC / 笔记本 客户端浏览器PDA / 手机 / 相机电子相册CRM云作为运算中心拍出的相片立即编辑修改在线编写文档、报告随时随地写日志随时随

4、地的身体健康状况监控musicpreferencesmapsnewscontactsmessagesmailing listsphotoe-mailscalendarphone numbersinvestmentsdatadevicesanywhereshared8企业云计算企业云计算from Salesforce91986年我国第一封Email 560bps 现在的网络速度 云计算的产生和演进云计算的产生和演进计算能力的需求的增长并行计算、分布式计算和网格计算并行计算Parallel Computing是指同时使用多种计算资源解决计算问题的过程，其主要目的是快速解决大型且复杂的计算问题特点：

5、把计算任务分派给系统内的多个运算单元大型机的多CPU和多存储器并行计算问题的特征将工作分离成离散部分，有助于同时解决随时并及时地执行多个程序指令（多条线同时运行）多计算资源下解决问题的耗时要少于单个计算资源下的耗时分布式计算Distributed Computing所谓分布式计算是一门计算机科学，它研究如何把一个需要非常巨大的计算能力才能解决的问题分成许多小的部分，然后把这些部分分配给许多计算机进行处理，最后把这些计算结果综合起来得到最终的结果。特点：把计算任务分派给网络中的多台独立的机器优点：稀有资源可以共享 通过分布式计算可以在多台计算机上平衡计算负载 可以把程序放在最适合运行它的计算机上

6、 分布式计算一些流行的分布式项目SETIHome：寻找外星文明RC-72：密码分析和破解，研究和寻找最为安全的密码系统Foldinghome：研究蛋白质折叠，误解，聚合及由此引起的相关疾病Rosettahome：蛋白质折叠项目，预测并设计蛋白质结构United Devices：寻找对抗癌症的有效的药物GIMPS：寻找最大的梅森素数（解决较为复杂的数学问题）网格计算Grid Computing网格是利用互联网把地理上广泛分布的各种资源（包括计算资源、存储资源、带宽资源、软件资源、数据资源、信息资源、知识资源等）连成一个逻辑整体，就像一台超级计算机一样，为用户提供一体化信息和应用服务（计算、存储、

7、访问等） 网格计算是分布式计算的一种，是分布式计算封装什么是云计算CLOUD COMPUTINGCloud Computing isNo softwareaccess everywhere by Internetpower - Large-scale data processingAppeal for startupsCost efficiencySoftware as a ServiceConsSecurityData lock-inSaaSPaaSUtility ComputingWhat Cloud Computing “IS NOT”? It is not Network Comput

8、ing Application and Data are not confined to any specific Companys Server No VPN Access Encompasses multiple companies, multiple servers and multiple networks It is not Traditional Outsourcing Not a contract to host data by 3rd party Hosting Business No subcontracting for computing services for spec

9、ific outside firmSo exactly what Cloud Computing is? A style of computing where massively scalable IT-enabled capabilities are provided as a service over the networkAcquisition Model Service BasedBusiness Model Usage BasedAccess Model NetworkTechnical Model Dynamic云计算云计算云计算是为用户提供无限计算资源的商业服务，是能够自我管理计

10、算资源的系统平台，是应用服务按需定制、易于扩展的软件架构。并行计算分布式计算网格计算CPU运算资源存储资源网络带宽云计算云计算（cloud computing），是一种基于互联网的计算方式，通过这种方式，共享的软硬件资源和信息可以按需提供给计算机和其他设备。22云计算相关概念云云是一些可以自我维护和管理的虚拟计算资源，通常为一些大型服务器集群，包括计算服务器、存储服务器、宽带资源等等。云计算将所有的计算资源集中起来，并由软件实现自动管理，无需人为参与。这使得应用提供者无需为繁琐的细节而烦恼，能够更加专注于自己的业务，有利于创新和降低成本。计算资源的演进：从集中到分散再到集中全世界只需要5台电脑

11、就足够了 托马斯沃森个人用户的内存只需640K足矣 比尔盖茨The network is the computer John Gage计算时代网络时代云时代Attributes of Cloud ComputingData stored on the cloudSoftware & services on the cloud - Access via web browserBased on standards and protocols - Linux, AJAX, LAMP, etc.Accessible from any deviceHardware CentricSoftware Cen

12、tricService CentricPersonal PCClient ServerCloud Computing云计算特点超大规模：服务器群虚拟化：可以看作是一片用于计算的云高可靠性：冗余副本、负载均衡通用性：支撑千变万化的实际应用高可扩展性：灵活、动态伸缩按需服务：按需购买极其廉价：不再需要一次性购买超级电脑安全： 摆脱数据丢失、病毒入侵 方便：支持多终端、数据共享云计算发展的障碍云计算目前的困境云计算何时从云端到地面标准不统一Google、Amazon、 IBM、微软等的平台互不兼容“云计算”之争 云计算何时从云端到地面数据真的安全？云服务提供商的信誉留后门？！面临着全世界的黑客需要高

13、强度的安全系统云计算何时从云端到地面网络带宽3G 尚未普及，费用极高云计算何时从云端到地面耗电量巨大主旋律节能减排终端设备的电池容量有限云计算的几大形式云计算服务类形基础设施即服务（ IaaS）软件即服务（ SaaS ）网络服务平台即服务（PaaS）管理服务提供商（MSP）商业服务平台云安全33提纲云计算概述云计算体系结构云计算关键技术 - 虚拟化云计算关键技术 - MapReduce云计算在北理34Cloud Computing Framework基础设施即服务 （实用计算、虚拟化）IaaS Infrastructure as a Service 是为IT行业创造虚拟的计算和数据中心，使得其

14、能够把计算单元、存储器、I/O设备、带宽等计算机基础设施，集中起来成为一个虚拟的资源池来为整个网络提供服务。用多少算多少Amazon WebServices，简作AWS弹性计算云EC2 （Elastic Compute Cloud） 计算简单存储服务S3 （Simple Storage Service） 存储RackspaceEucalyptusGoogleWhat are the benefits & challenges IaaS?BenefitsSystems managed by SLA should equate to fewer breaches Higher return on

15、assets through higher utilizationReduced cost driven byLess hardwareLess floor space from smaller hardware footprintHigher level of automation from fewer administratorsLower power consumptionAble to match consumption to demand ChallengesPortability of applicationsMaturity of systems management tools

16、Integration across the Cloud boundaryExtension of internal security models软件即服务SaaS Software as a ServiceSaaS是一种基于互联网提供软件服务的应用模式。软件租赁：用户按使用时间和使用规模付费绿色部署：用户不需安装，打开浏览器即可运行不需要额外的服务器硬件软件（应用服务）按需定制软件即服务SaaS 产品Salesforce CRM阿里软件 Google appsAlexa 排名：第一名 Salesforce第二名 阿里软件第三名 铭万第四名 金算盘第五名 中企动力第六名 神码在线第七名 商务

17、领航第八名 友商网第九名 八百客第十名 What are the benefits & challenges of SaaS?BenefitsSpeedReduced up-front cost, potential for reduced lifetime costTransfer of some/all support obligationsElimination of licensing riskElimination of version compatibilityReduced hardware footprintChallengesExtension of the security

18、 model to the provider (data privacy and ownership)Governance and billing managementSynchronization of client and vendor migrationsIntegrated end-user supportScalabilityStrong governance required to prevent lines of business from purchasing application services externally without IT involvement平台即服务

19、PaaS Platform as a Service把服务器平台或开发环境作为一种服务提供的商业模式从系统定制到PaaS 的 800app 不再需要任何编程即可开发包括CRM、OA、HR、SCM、进销存管理等任何企业管理软件What are the benefits & challenges of PaaS?BenefitsPay-as-you-go for development, test, and production environmentsEnables developers to focus on application codeInstant global platformEli

20、mination of H/W dependencies and capacity concernsInherent scalabilitySimplified deployment modelChallengesGovernanceTie-in to the vendorExtension of the security model to the providerConnectivityReliance on 3rd party SLAsStrong governance required to prevent lines of business from building applicat

21、ions without IT involvementSolutions and vendors are emerging dailyExternal IaaSUtility Systems Management Tools+Utility Application DevelopmentData SynapseUniva UDElastra Cloud Server3tera App LogicVMWareIBM TivoliCassattParallelsHP/EDS (TBD)IBM Blue CloudSun GridJoyentSoftware as a Service (Saas)G

22、oogle AppsZoho OfficeWorkdayMicrosoft Office LivePlatform as a ServiceAmazon E2CS FGoogle App EngineCogheadInternal IaaSHP Adaptive Infrastructure as a ServiceOracle On Demand AppsNetSuite ERPS SFAEtelosLongJumpBoomiMicrosoft Azure*XenZuoraAria SystemseVaptIBM WebSphere XD BEA Weblogic Server VEMule

23、RackspaceJamcracker43提纲云计算概述云计算体系结构云计算关键技术 - 虚拟化云计算关键技术 - MapReduce云计算在北理Role of OS44传统服务器45Web ServerWindowsIISApp ServerLinuxGlassfishDB ServerLinuxMySQLEMailWindowsExchange虚拟服务器46Virtual Machine Monitor (VMM) layer between Guest OS and hardware 虚拟化47虚拟化历史481965 IBM M44/44X paging system1965 IBM S

24、ystem/360-67 virtual memory hardware1967 IBM CP-40 (January) and CP-67 (April) time-sharing1972 IBM VM/370 run VM under VM 1997 Connectix First version of Virtual PC 1998 VMWare U.S. Patent 6,397,2421999 VMware Virtual Platform for the Intel IA-32 architecture2000 IBM z/VM2001 Connectix Virtual PC f

25、or Windows2003 Microsoft acquired Connectix 2003 EMC acquired Vmware2003 VERITAS acquired Ejascent 2005 HP Integrity Virtual Machines 2005 Intel VT2006 AMD VT2005 XEN2006 VMWare Server2006 Virtual PC 20062006 HP IVM Version 2.02006 Virtual Iron 3.12007 InnoTek VirtualBox2007 KVM in Linux Kernel2007

26、XEN in Linux Kernel虚拟化Virtualization:The ability to run multiple operating systems on a single physical system and share the underlying hardware resources49Low utilization metrics in servers across the organizationToo many servers for too little workHigh costs and infrastructure needsMaintenanceLeas

27、esNetworkingFloor spaceCoolingPowerDisaster RecoveryHeterogeneous Environments动态迁移在不中断服务的情况下，将VM迁移到其他的物理服务器上分区在单一物理服务器上同时运行多个虚拟机隔离在同一服务器上的虚拟机之间相互隔离封装整个虚拟机都保存在文件中，而且可以通过移动和复制这些文件的方式来移动和复制该虚拟机虚拟技术: 四大特性动态迁移通过动态地将应用程序从一个服务器移动到另一个服务器，减少计划停机时间。通过允许您将工作负载从负载较重的服务器移动到具有空闲容量的服务器，可以应对不断变化的工作负载和业务需求。通过允许您简单地整

28、合工作负载，并关闭不使用的服务器，减少能量的消耗。54定义Hypervisor (or VMM Virtual Machine Monitor) is a software layer that allows several virtual machines to run on a physical machineThe physical OS and hardware are called the HostThe virtual machine OS and applications are called the Guest55虚拟化架构VMware ESX, Microsoft Hyper

29、-V, XenHardwareHypervisorVM1VM2Type 1 (bare-metal)HostGuestHardwareOSProcessHypervisorVM1VM2Type 2 (hosted)VMware Workstation, Microsoft Virtual PC, Sun VirtualBox, QEMU, KVMHostGuestBare-metal architecture-裸金属虚拟化结构Hosted architecture -主机虚拟化结构虚拟化类型57完全虚拟化半虚拟化操作系统层虚拟化全虚拟化Guest os 内核不需要进行修改。Guest Doma

30、in不知道自己运行在Hypervisor上完全虚拟化技术的优点是有很好的兼容性，操作系统不用改动就能安装到虚拟服务器上；主要缺点是，hypervisor给处理器带来开销。全虚拟化准虚拟化 guest os 内核需要修改，能够与hypervisor协同工作。当Guest Domain是一个准虚拟化的虚拟机时，虚拟机的内核是被修改过的，它知道自己不是运行在真实的硬件上。其速度能力几乎不亚于未经过虚拟化处理的服务器；缺点是只适用于BSD、Linux、Solaris等某些开源操作系统，不适用于Windows等专有操作系统，兼容性差。准虚拟化操作系统层虚拟化 操作系统层虚拟化没有独立hypervisor

31、层，主机操作系统本身负责在多个虚拟服务器之间分配硬件资源，并且让这些服务器彼此独立。操作系统层虚拟化的缺点是所有虚拟服务器必须运行同一操作系统(不过每个实例有各自的应用程序和用户账户)，灵活性比较差；优点是本机速度性能比较高，由于架构在所有虚拟服务器上使用单一、标准的操作系统，管理起来比异构环境要容易。kvmXen 3.0Available from Xen Source ()In association with University of Cambridge (http:/www.cl.cam.ac.uk/Research/SRG/netos/xen/)Support for 64-Bit

32、 and 32-bit machinesSupports IntelVTLinux support only, Windows expected later this yearOpen Source Product One of the most actively maintained projects in the open source community$ - FreeTarget: 100 virtual OSes per machineXen ArchitectureDomain 0Domain UHypervisor66提纲云计算概述云计算体系结构云计算关键技术 - 虚拟化云计算关

33、键技术 - MapReduce云计算在北理海量信息处理你需要一个多大的硬盘？6869How much data?Internet archive has 2 PB of data + 20 TB/monthGoogle processes 20 PB a day (2008)“all words ever spoken by human beings” 5 EBCERNs LHC will generate 10-15 PB a yearSanger anticipates 6 PB of data in 2009640K ought to be enough for anybody.What

34、 is MapReduce?Data-parallel programming model for clusters of commodity machinesPioneered by GoogleProcesses 20 PB of data per dayPopularized by open-source Hadoop projectUsed by Yahoo!, Facebook, Amazon, What is MapReduce Used For?At Google:Index building for Google SearchArticle clustering for Goo

35、gle NewsStatistical machine translationAt Yahoo!:Index building for Yahoo! SearchSpam detection for Yahoo! MailAt Facebook:Data miningAd optimizationSpam detectionExample: Facebook LexiconExample: Facebook LexiconWhat is MapReduce Used For?In research:Analyzing Wikipedia conflicts (PARC)Natural lang

36、uage processing (CMU) Bioinformatics (Maryland)Particle physics (Nebraska)Ocean climate simulation (Washington)MapReduce GoalsScalability to large data volumes:Scan 100 TB on 1 node 50 MB/s = 24 daysScan on 1000-node cluster = 35 minutesCost-efficiency:Commodity nodes (cheap, but unreliable)Commodit

37、y networkAutomatic fault-tolerance (fewer admins)Easy to use (fewer programmers)Typical Hadoop Cluster40 nodes/rack, 1000-4000 nodes in cluster1 Gbps bandwidth in rack, 8 Gbps out of rackNode specs (Facebook):8 cores, 16 GB RAM, 8 x 1.5 TB disks, no RAIDAggregation switchRack switchTypical Hadoop Cl

38、usterGoogle Server78ChallengesCheap nodes fail, especially if you have manyMean time between failures for 1 node = 3 yearsMTBF for 1000 nodes = 1 daySolution: Build fault-tolerance into systemCommodity network = low bandwidthSolution: Push computation to the dataProgramming distributed systems is ha

39、rdSolution: Users write data-parallel “map” and “reduce” functions, system handles work distribution and failuresHadoop ComponentsDistributed file system (HDFS)Single namespace for entire clusterReplicates data 3x for fault-toleranceMapReduce frameworkRuns jobs submitted by usersManages work distrib

40、ution & fault-toleranceColocated with file systemHadoop Distributed File SystemFiles split into 64MB blocksBlocks replicated across several datanodes (usually 3)Namenode stores metadata (file names, locations, etc)Optimized for large files, sequential readsFiles are append-onlyNamenodeDatanodes12341

41、24213143324File1MapReduce82“Work”w1w2w3r1r2r3“Result”“worker”“worker”“worker”PartitionCombineMapReduce Programming ModelData type: key-value recordsMap function:(Kin, Vin) list(Kinter, Vinter)Reduce function:(Kinter, list(Vinter) list(Kout, Vout)Parallel/Distributed Computing Programming ModelInput

42、split shuffle output MapReduce Programming Model读入数据: key/value 对的记录格式数据Map: 从每个记录里extract somethingmap (in_key, in_value) - list(out_key, intermediate_value) 处理input key/value pair 输出中间结果key/value pairsShuffle: 混排交换数据把相同key的中间结果汇集到相同节点上Reduce: aggregate, summarize, filter, etc.reduce (out_key, list

43、(intermediate_value) - list(out_value) 归并某一个key的所有values，进行计算输出合并的计算结果 (usually just one) 输出结果MapReduce Programming Model86Word Frequencies in Web pages输入：one document per record用户实现map function，输入为key = document URLvalue = document contentsmap输出 (potentially many) key/value pairs. 对document中每一个出现的词

44、，输出一个记录87Example continued:MapReduce运行系统(库)把所有相同key的记录收集到一起 (shuffle/sort)用户实现reduce function对一个key对应的values计算求和sumReduce输出 88MapReduce Runtime SystemExample: Word Countdef mapper(line): foreach word in line.split(): output(word, 1)def reducer(key, values): output(key, sum(values)Word Count Executionthe quickbrown foxthe fox atethe mousehow nowbrown cowMapMapMapReduceReducebrown, 2fox, 2how, 1now, 1the, 3ate, 1cow, 1mouse, 1quick, 1the, 1brown, 1fox, 1quick, 1the, 1fox, 1the, 1how, 1now, 1brown, 1ate, 1mouse, 1cow