网格计算与云计算

刘鹏gridcloud 中国网格信息中转站 中国云计算 网格计算与云计算 GridComputing CloudComputing 报告简介 自2007年第4季度开始 云计算 变成了炙手可热的一个词 Google Amazon IBM 微软等IT巨头们以前所未有的速度和规模推动云计算技术和产品的普及 一些学术活动迅速将云计算提上议事日程 支持的声音和反对的声音不绝于耳 那么 云计算到底是什么 它是不是一种炒作 从技术上看它是如何实现的 它与网格计算是什么关系 网格计算搞了这么多年 结果如何 网格计算和云计算的未来是什么 本次讲座试图基于大量的实例和数据分析这些问题 主题报告 网格计算与云计算报告人 刘鹏教授报告时间 2009年3月6日 周五 9 00 11 00报告地点 上海大学延长校区行健楼707报告厅 标志此页从他人的PPT引用而来 标志此页从他人的PPT改造而来 著作权声明 本PPT相当部分页面引用或改造自他人PPT 其著作权属于其创作者 在此 向他们在网上共享所创作的内容表示致敬和感谢 内容提纲 云计算是个新名词 CloudComputing 5 Google云计算 Google云计算数据中心 UserCentric Datastoredinthe Cloud Datafollowsyou yourdevicesDataaccessibleanywhereDatacanbesharedwithothers Google的三大法宝 GoogleFileSystem GFS BigTableMapReduce 10 GFSArchitecture Google48 MSN19 Yahoo33 Filesbrokenintochunks typically64MB MastermanagesmetadataDatatransfershappendirectlybetweenclients chunkservers Client Replicas Masters GFSMaster GFSMaster C0 C1 C2 C5 Chunkserver1 C0 C2 C5 ChunkserverN C1 C3 C5 Chunkserver2 GFSUsage Google 200 clustersFilesystemclustersofupto5000 machinesPoolsof10000 clients5 PetabyteFilesystemsAllinthepresenceoffrequentHWfailure Google的三大法宝 GoogleFileSystem GFS BigTableMapReduce BigTable Datamodel row column timestamp cellcontents BigTable Distributedmulti levelsparsemapFault tolerance persistentScalableThousandofserversTerabytesofin memorydataPetabytesofdisk baseddataSelf managingServerscanbeadded removeddynamicallyServersadjusttoloadimbalance Google的三大法宝 GoogleFileSystem GFS BigTableMapReduce MapReduce ANewModelandSystem TwophasesofdataprocessingMap in key in value keyj valuej j 1 k Reduce key value1 valuem key f value 亚玛逊云计算 AmazonSimpleStorageService 15perGBpermonthstorage Object BasedStorage1B 5GB objectFast Reliable ScalableRedundant Dispersed99 99 AvailabilityGoalPrivateorPublicPer objectURLs ACLsBitTorrentSupport 10 18perGBdatatransfer 01for1000to10000requests S3SOAP QueryAPI Service ListAllMyBucketsBuckets CreateBucketDeleteBucketListBucketGetBucketAccessControlPolicySetBucketAccessControlPolicyGetBucketLoggingStatusSetBucketLoggingStatusObjects PutObjectPutObjectInlineGetObjectGetObjectExtendedDeleteObjectGetObjectAccessControlPolicySetObjectAccessControlPolicy AmazonElasticComputeCloud 10perserverhour VirtualComputeCloudElasticCapacity1 7GHzx861 7GBRAM160GBDisk250MB SecondNetworkNetworkSecurityModelTimeorTraffic basedScaling Loadtesting SimulationandAnalysis Rendering SoftwareasaServicePlatform Hosting 10 18perGBdatatransfer EC2SOAP QueryAPI Images RegisterImageDescribeImagesDeregisterImageInstances RunInstancesDescribeInstancesTerminateInstancesGetConsoleOutputRebootInstancesKeypairs CreateKeyPairDescribeKeyPairsDeleteKeyPair ImageAttributes ModifyImageAttributeDescribeImageAttributeResetImageAttributeSecurityGroups CreateSecurityGroupDescribeSecurityGroupsDeleteSecurityGroupAuthorizeSecurityGroupIngressRevokeSecurityGroupIngress Web ScaleArchitecture 微软云计算 34 35 36 EnterpriseITspendingchallenge Source IBMCorporateStrategyanalysisofIDCdata Sept 2007 GlobalAnnualITSpendingEstimatedUS B1996 2010 0B 50 100 150 200 250 300 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 NewServerSpending ServerMgtandAdminCosts PowerandCoolingCosts DreamorNightmare 新概念层出不穷 UtilityComputing SaaS SoftwareasaService GridComputing ClusterComputing CloudComputing 可以分为两类 使用模式 基础设施 云计算的技术背景 云计算是并行计算 ParallelComputing 分布式计算 DistributedComputing 和网格计算 GridComputing 的发展 或者说是这些计算机科学概念的商业实现 云计算是虚拟化 Virtualization 效用计算 UtilityComputing IaaS 基础设施即服务 PaaS 平台即服务 SaaS 软件即服务 等概念混合演进并跃升的结果 云计算的定义 云计算是一种商业计算模型 它将计算任务分布在大量计算机构成的资源池上 使各种应用系统能够根据需要获取计算力 存储空间和各种软件服务 43 云计算的概念模型 云计算的服务类型 云计算技术体系结构 云计算简化实现机制 云计算的特点 超大规模 虚拟化 高可靠性 通用性 高可扩展性 按需服务 极其廉价 内容提纲 网格计算的定义 在动态变化 由多个机构组成的虚拟组织中协调资源共享和求解问题 51 虚拟组织 TeraGrid 53 WhatistheTeraGrid Technology Support Science NSF已投资2 460亿美元自2004年10月已处于生产运行阶段 目前已用高性能网络集成了每秒750万亿次计算能力 30PB存储空间和100多个学科的数据库资源 54 TeraGridPI sByInstitution TeraGridPI s Blue 10ormorePI sRed 5 9PI sYellow 2 4PI sGreen 1PI 55 TeraGridUsed 56 TeraGrid s3 prongedstrategytofurtherscience DEEPScience EnablingTerascaleScienceMakesciencemoreproductivethroughanintegratedsetofvery highcapabilityresourcesASTAprojectsWIDEImpact EmpoweringCommunitiesBringTeraGridcapabilitiestothebroadsciencecommunityScienceGatewaysOPENInfrastructure OPENPartnershipProvideacoordinated generalpurpose reliablesetofservicesandresourcesGridinteroperabilityworkinggroup 57 ScienceGatewaysAnewinitiativefortheTeraGrid Provide TeraGridInside capabilitiesApplicationprogramsrunningonusers machinesbutaccessingservicesinTeraGrid WorkflowComposer 58 Gatewaysaregrowinginnumbers 10initialprojectsaspartofTGproposal 20GatewayprojectstodayNolimitonhowmanygatewayscanuseTGresourcesPrepareservicesanddocumentationsodeveloperscanworkindependently OpenScienceGrid OSG SpecialPRiorityandUrgentComputingEnvironment SPRUCE NationalVirtualObservatory NVO LinkedEnvironmentsforAtmosphericDiscovery LEAD ComputationalChemistryGrid GridChem ComputationalScienceandEngineeringOnline CSE Online GEON GEOsciencesNetwork NetworkforEarthquakeEngineeringSimulation NEES SCECEarthworksProjectNetworkforComputationalNanotechnologyandnanoHUBGIScienceGateway GISolve BiologyandBiomedicineScienceGatewayOpenLifeSciencesGatewayTheTelescienceProjectGridAnalysisEnvironment GAE NeutronScienceInstrumentGatewayTeraGridVisualizationGateway ANLBIRNGridblastBioinformaticsGatewayEarthSystemsGridAstrophysicalDataRepository Cornell ManyothersinterestedSIDGridHASTAC OpenScienceGrid 60 96Resourcesacrossproduction integrationinfrastructures 20VirtualOrganizations 6operationsIncludes25 non physics 20 000CPUs from30to4000 6PBTapes 4PBSharedDisk SnapshotofJobsonOSGs SustainingthroughOSGsubmissions 3 000 4 000simultaneousjobs 10Kjobs day 50KCPUhours day Peaktestjobsof15Kaday Usingproduction researchnetworks OSGSnapshot NERSC BU UNM SDSC UTA OU FNAL ANL WISC BNL VANDERBILT PSU UVA CALTECH IOWASTATE PURDUE IU BUFFALO TTU CORNELL ALBANY UMICH INDIANA IUPUI STANFORD UWM UNL UFL KU UNI WSU MSU LTU LSU CLEMSON MCGILL UMISS UIUC UCR UCLA LEHIGH NSF ORNL HARVARD UIC SMU UCHICAGO WhatistheOpenScienceGrid Brazil Mexico Tawain UK OSG应用 Genomesequenceanalysis STAR 5TBtransfer SRM GridFTP Sloandigitalskysurvey EarthSystemGrid O 100TB onlinedata EarthSystemGrid Sloandigitalskysurvey Chandra Hubble MMT Sub mmarray VLA Antarticasub mm Magellan6 5m Whippleg ray SIRTF OakRidge 1 2mCO LAMOST Sloandigitalskysurvey EGEE EnablingGridsforE sciencE EuropeanGridInitiative June2 2008 68 ArcheologyAstronomyAstrophysicsCivilProtectionComp ChemistryEarthSciencesFinanceFusionGeophysicsHighEnergyPhysicsLifeSciencesMultimediaMaterialSciences 250sites48countries 50 000CPUs 20PetaBytes 10 000users 150VOs 150 000jobs day EGEE EGEEworkloadin2007 CPU 114Millionhours Data 25PBstored11PBtransferred LCG LHCComputingGrid LHC LargeHadronicCollider GRIDTutorial Howto