网格计算的趋势、挑战、技术和应用英文版

1、Introduction to Grid Computing:Trends, Challenges, Technologies, Applications Dr. Rajkumar Buyya Cloud Computing and Distributed Systems (CLOUDS) LaboratoryDept. of Computer Science and Software EngineeringThe University of Melbourne, AustraliaOutlineIntroductionUtility Networks, Trends, and Grid Co

2、mputingDefining GridsWhat is Grid?, Types of Grid Services Global Grids and ChallengesCharacteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Science and e-BusinessConclusion4 Essential

3、 Utilities and Delivery Networks(1) WaterWater Distribution Network(4) TelephoneTelecom Networks(3) Gas(2) ElectricityPower Grid(5) Computing: Grid Delivery IT services as the 5th utility (Power Grid inspiration)eScienceeBusinesseGovernmenteHealthMultilingualeEducationPower Grid Inspiration: Seamles

4、sly delivering electricity as a utility to users Why Grid Computing Now?Let us look at the Evolution of ICTGrid-like VisionIn 1969, Leonard Kleinrock, one of the chief scientists of the original ARPA project which seeded the Internet, wrote:As of now, computer networks are still in their infancy, bu

5、t as they grow up and become sophisticated, we will probably see the spread of computer utilities, which, like present electric and telephone utilities, will service individual homes and offices across the country“Despite major advances in hardware and software systems over the past 40 years, we are

6、 yet to realise this vision. How far are we still from delivering computing as a utility?Let us look into the ICT evolution and project the future.Computing and Communication Technologies Evolution: 1960-2010!* Sputnik19601970197519801985199019952000* ARPANET* Email* Ethernet* TCP/IP* IETF* Internet

7、 Era* WWW Era* Mosaic* XML* PC Clusters*Crays*MPPs* Mainframes* HTML* W3C* P2P* Grids* XEROX PARC wormCOMPUTINGCommunication* Web Services* Minicomputers* PCs* WS Clusters* PDAs* Workstations* HTC2010* e-Science* Computing as Utility* e-Business* SocialNetControlCentralisedDecentralised2100210021002

8、10021002100210021002100Personal DeviceSMPs or SuperComputersLocalClusterGlobalGridSERV ICES+PERFORMANCEInter PlanetGridIndividualGroupDepartmentCampusStateNationalGlobeInter PlanetUniverseAdministrative BarriersEnterpriseCluster/GridComputing is Scaling: Towards Inter-Planetary LevelOutlineIntroduct

9、ionUtility Networks, Trends, and Grid ComputingDefining GridsWhat is Grid?, Types of Grid Services Global Grids and ChallengesCharacteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Sci

10、ence and e-BusinessConclusionWhat does Grid mean?/ (It means different things to different people)IBMOn Demand ComputingMicrosoft.NETOracle10gSunN1 Sun Grid EngineHPAdaptive EnterpriseAmazonElastic Compute Cloud ServicesUnited Devices and related companies:Harvesting Unused Desktop resourcesWhat is

11、Grid?Buyya et. alA type of parallel and distributed system that enables the sharing, exchange, selection, & aggregation of geographically distributed “autonomous” resources:Computers PCs, workstations, clusters, supercomputers, laptops, notebooks, mobile devices, PDA, etc;Software e.g., ASPs renting

12、 expensive special purpose applications on demand;Catalogued data and databases e.g. transparent access to human genome database;Special devices/instruments e.g., radio telescope SETIHome searching for life in galaxy.People/collaborators.depending on their availability, capability, cost, and user Qo

13、S requirements.WideareaThe Grid(Foster et. al) “Resource sharing & coordinated problem solving in dynamic, multi-institutional virtual organizations”Enable integration of distributed resourcesUsing general-purpose protocols & infrastructureTo achieve better-than-best-effort serviceHow does Grids loo

14、k like?A Bird Eye View of a Global GridGrid Resource BrokerResource BrokerApplicationGrid Information ServiceGrid Resource BrokerdatabaseR2R3RNR1R4R5R6Grid Information ServiceHow Are Grids Used?High-performance computingCollaborative data-sharingCollaborative designDrug discoveryFinancial modelingDa

15、ta center automationHigh-energy physicsLife sciencesE-BusinessE-ScienceNatural language processing & Data MiningUtility computingClasses of Grid Services / Types of GridsComputational Services CPU cyclesPooling computing power: SETIHome, TeraGrid, AusGrid, ChinaGrid, IndiaGrid, UK Grid, Data Service

16、sCollaborative data sharing generated by instruments, sensors, persons: LHC Grid, NapsterApplication ServicesAccess to remote software/libraries and license managementNetSolve Interaction ServiceseLearning, Virtual Tables, Group Communication (Access Grid), GamingKnowledge ServicesThe way knowledge

17、is acquired, processed and manageddata mining.Utility Computing ServicesTowards a market-based Grid computing: Leasing and delivering Grid services as ICT utilities.Computational GridData GridASP GridInteraction GridKnowledge GridUtility GridinfrastructureUsersWorldwide Grid Spending: 2005 to 2010 i

18、n billion dollars After the year 2007, business popularity of Grid computing is expected to accelerate:Especially, the financial services and ERP services is expected to take major parts in the expenseSource: Insight Research Corp.BillionsOutlineIntroductionUtility Networks, Trends, and Grid Computi

19、ngDefining GridsWhat is Grid?, Types of Grid Services Global Grids and ChallengesCharacteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Science and e-BusinessSummary and ConclusionGrid

20、 & Related ParadigmsUtility ComputingComputing “services”No knowledge of providerEnabled by grid technologyDistributed ComputingLoosely coupledHeterogeneousSingle AdministrationClusterTightly coupledHomogeneousCooperative workingGrid ComputingLarge scaleCross-organizationalGeographical distributionD

21、istributed ManagementHiro Kishimoto Some Characteristics of GridsNumerousresourcesDifferent securityrequirements & policiesResources areheterogeneousGeographicallydistributedDifferent resourcemanagementpoliciesConnected byheterogeneous, multi-level networksOwned by multiple organizations & individua

22、lsUnreliable resources and environmentsSlide by HiroGrid ChallengesSecurityResource Allocation & SchedulingData localityNetwork ManagementSystem ManagementResource DiscoveryUniform AccessComputational EconomyApplication ConstructionSome Grid Initiatives WorldwideAustraliaNimrod-GGridbusGrangeNet.APA

23、CGridARC eResearchBrazilOurGrid, EasyGridLNCC-Grid + many othersChinaChinaGrid EducationCNGrid - applicationEuropeUK eScienceEU Grids.and many more.IndiaGarudaJapanNAREGIKorea.N*GridSingaporeNGPUSAGlobusGridSecAccessGridTeraGridCyberinfrastureand many more.Industry InitiativesIBM On Demand Computing

24、HP Adaptive ComputingSun N1Microsoft - .NETOracle 10gInfosys Enterprise GridSatyam Business GridStorageTek Grid.and many morePublic ForumsOpen Grid ForumAustralian Grid ForumConferences:CCGridGridHPDCE-Science1.3 billion 3 yrs1 billion 5 yrs450million 5 yrs486million 5 yrs1.3 billion (Rs)27 million2

25、? billion 120million 5 yrsmix-and-match Object-orientedInternet/partial-P2PGrid ComputingApproachesNetwork enabled SolversNetSolveEconomic-based Utility / Service-Oriented ComputingNimrod-GOpen-Source Grid Middleware ProjectsOutlineIntroductionUtility Networks, Trends, and Grid ComputingDefining Gri

26、dsWhat is Grid?, Types of Grid Services, Global Grids and ChallengesCharacteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Science and e-BusinessConclusionGrid Realization Steps/Requir

27、ementsStep 1:The integration of individual s/w & h/w components into a combined networked resource (single system image cluster).Step 2:Low-level middleware to provide a secure and uniform access to services provided by different resources.Step 3:User-level middleware to support application developm

28、ent and aggregation of distributed resources.Step 4:The construction of Grid applications and Web portals.Networked Resources across OrganizationsComputersNetworksData SourcesScientific InstrumentsStorage SystemsLocal Resource ManagersOperating SystemsQueuing SystemsInternet ProtocolsLibraries & App

29、 KernelsDistributed Resources Coupling ServicesInformationQoSProcessApplication Development and Deployment EnvironmentLanguages/CompilersLibrariesDebuggersWeb toolsResource Management and Scheduling:Applications and PortalsProb. Solving Env.ScientificCollaborationEngineeringWeb enabled AppsTradingFA

30、BRICAPPLICATIONSSECURITY LAYERSecurityDataCORE MIDDLEWAREUSER LEVEL MIDDLEWAREMonitorsLayered Grid ArchitectureAdaptive Management Autonomic/ Grid EconomyOutlineIntroductionUtility Networks, Trends, and Grid ComputingDefining GridsWhat is Grid?, Types of Grid Services Global Grids and ChallengesChar

31、acteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Science and e-BusinessConclusionGrid Middleware TechnologiesGlobus Argonne National Lab and ISIGridbus University of MelbourneUnicore

32、 European Project (Germany)Legion University of VirginiaGlobus Toolkit ServicesSecurity (GSI)PKI-based Security (Authentication) ServiceJob submission and management (GRAM)Uniform Job SubmissionInformation services (MDS)LDAP-based Information ServiceRemote file management (GASS)Remote Storage Access

33、 Service Remote Data Catalogue and Management ToolsRecently move to:WSRF (Web Services Resource Framework)GRAM ComponentsGlobus SecurityInfrastructureJob ManagerGRAM client API calls to request resource allocationand process creation.MDS client API callsto locate resourcesQuery current statusof reso

34、urceCreateRSL LibraryParseRequestAllocate &create processesProcessProcessProcessMonitor &controlSite boundaryClientMDS: Grid Index Info ServerGatekeeperMDS: Grid Resource Info ServerLocal Resource ManagerMDS client API callsto get resource infoGRAM client API statechange callbacksSample of High-Leve

35、l ServicesResource brokers and co-allocatorsDUROC, Nimrod-G, Gridbus, Condor-G, AppLeS PSTCommunication & I/O librariesMPICH-G, RIO (MPI-IO)Parallel languagesHPC+, CC+Collaborative environmentsCAVERNsoft, ManyWorldsOthersMetaNEOS, NetSolve, LSA, AutoPilot, WebFlowA resource broker for managing, stee

36、ring, and executing task farming (parameter sweep/SPMD model) applications on the Grid based on deadline and computational economy. Based on users QoS requirements, our Broker dynamically leases services at runtime depending on their quality, cost, and availability. Key FeaturesA single window to ma

37、nage & control experimentPersistent and Programmable Task Farming EngineResource DiscoveryResource Trading Scheduling & PredicationsGeneric Dispatcher & Grid AgentsTransportation of data & resultsSteering & data managementAccountingUses Globus MDS, GRAM, GSI, GASSThe Nimrod-G Grid Resource Broker(fr

38、om Monash University)Condor-G: Condor for the Grid(from University of Wisconsin)Condor is a high-throughput schedulerCondor-G uses Globus Toolkit libraries for:Security (GSI)Managing remote jobs on Grid (GRAM)File staging & remote I/O (GSI-FTP)Grid job management interface & schedulingRobust replace

39、ment for Globus Toolkit programsGlobus Toolkit focus is on libraries and services, not end user vertical solutionsSupports single or high-throughput apps on GridPersonal job manager which can exploit Grid resourcesThe Gridbus Project Melbourne:Enable Leasing of ICT Services on DemandWWGPushes Grid c

40、omputing into mainstream computing Gridbus Gridbus Architecture LayerAdaptive Management On Demand Assembly of Services: Putting Them All TogetherASP CatalogueGrid Info ServiceGrid Market DirectoryGSP(Accounting Service)GridbusGridBankGSP(e.g., UofM)PEGSP(e.g., VPAC)PEGSP(e.g., IBM)CPUorPEGrid Servi

41、ce (GS)(Globus)AlchemiGSGTSCluster SchedulerJob8GridResource Broker2Visual Application ComposerApplication CodeExplore data13645Results97Results+Cost Info1011Bill12Data CatalogueAlchemi: .NET-based Enterprise Grid Platform & Web ServicesInternetInternetAlchemi Worker AgentsAlchemi ManagerAlchemi Use

42、rsWeb ServicesSETIHome like ModelGeneral PurposeDedicated/Non-dedicate workersRole-based Security.NET and Web ServicesC# ImplementationGridThread and Job Model ProgrammingEasy to setup and use Widely in use!Some Users of AlchemiTier Technologies, USALarge scale document processing using Alchemi fram

43、ework CSIRO, AustraliaNatural Resource ModelingThe Friedrich Miescher Institute (FMI) for Biomedical Research, SwitzerlandPatterns of transcription factors in mammalian genes Satyam Computers Applied Research Laboratory, IndiaMicro-array data processing using Alchemi framework The University of Sao

44、Paulo, BrazilThe Alchemi Executor as a Windows Service stochastix GmbH, GermanyServing clients in International Banking/Finance sectorMany users in Universities: See next for an example.Gridbus Broker and Remote Service Access EnablersAlchemiGatewayUnicoreData StoreAccess TechnologyGrid FTPSRB-PBS-C

45、ondor-SGEGlobusJob managerfork()batch()GridbusagentData Catalog-PBS-Condor-SGE-XGridSSHfork()batch()GridbusagentCredential RepositoryMyProxyHome Node/PortalGridbusBrokerfork()batch()-PBS-Condor-SGE-Alchemi-XGridPortletsComparison of Middleware Technologies MiddlewarePropertyUNICOREGlobusLegionGridbu

46、sFocusHigh level Programming modelsLow level servicesHigh level Programming modelsAbstractions and market modelsCategoryMainly uniform job submission and monitoring Generic computational Generic computational Generic computationalArchitecture Vertical multi tiered systemLayered and modular toolkitVe

47、rtically integrated systemLayered component and utility modelImplementation Model Abstract Job Object Hourglass model at system level Object-oriented metasystem Hourglass model at user level Implementation TechnologiesJava C and Java C+C, Java, C# and PerlRuntime PlatformUnixUnixUnixUnix and Windows

48、 with .NETProgramming EnvironmentWorkflow environmentReplacement libraries for Unix & C libraries. Special MPI library (MPICH G), CoG (Commodity Grid) kits in Java, Python, CORBA, Matlab, Java Server Pages, Perl and Web Services Legion Application Programming Interfaces (API).Command line utilities

49、Broker Java APIXML-based parameter-sweep language Grid Thread model via Alchemi.Some Users and ApplicationsEuroGrid, Grid Interoperability Project, OpenMolGrid and Japanese NAREGI.AppLeS, Ninf, Nimrod-G, NASA IPG, Condor-G, Gridbus Broker, UK eScience Project, GriPhyN, and EU Data Grid.NPACI Testbed

50、, Nimrod-L, and NCBioGrid. Additionally, it has been used in the study of axially symmetric steady flow and protein folding applications.ePhysics, Belle Analysis Data Grid, NeuroGrid, Natural Language Engineering, HydroGrid, and Amsterdam Private Grid.OutlineIntroductionUtility Networks, Trends, and

51、 Grid ComputingDefining GridsWhat is Grid?, Types of Grid Services Global Grids and ChallengesCharacteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Science and e-BusinessConclusionCol

52、laborative Science and Global Grid Distributed instruments Distributed computation Distributed data Peers sharing ideas and collaborative interpretation of data/results 2100 2100 2100 2100 2100 2100 2100 2100 Remote Visualization Data & Compute Service Global GridE-ScientistSome e-Science Grid Appli

53、cations Bioinformatics: Drug Design / Protein ModellingSensitivityexperiments on smog formationNatural Language EngineeringEcological Modelling: Control Strategies for Cattle TickElectronic CAD: Field Programmable Gate ArraysComputer Graphics: Ray TracingHigh Energy Physics: Searching for Rare Event

54、sFinance: Investment Risk AnalysisVLSI Design: SPICE SimulationsAerospace: Wing DesignNetwork SimulationAutomobile:Crash Simulation Data MiningCivil Engineering:Building Design astrophysics 1. Grid Use in Science Online Medical Instrumentation and NeuroscienceOsaka Univ. HospitalOsaka Univ.DV transf

55、erLife-electronics laboratory,AISTData AnalysisProvision of MEGProvision of expertise in the analysis of brain functionCybermedia CenterData GenerationAnalysis ResultsAnalysis ResultsAVirtual Laboratoryfor medicine and brain scienceKnowledge sharingMEG sharing?Data Sharing2: Grid in Data Sharing Phy

56、sics Collaboration (fundamental investigation on the origin of mass)LHC Grid Computing ModelTier2 Centre 1 TIPSOnline SystemOffline Processor Farm 20 TIPSCERN Computer CentreAsia Pacific Centre 4 TIPSFrance Regional Centre US Regional Centre Italy Regional Centre InstituteInstituteInstitutePhysicist

57、 desktop computers100 MBytes/sec100 MBytes/sec622 Mbits/sec10 to 100 Mbits/secStorageStorageThere is a “bunch crossing” every 25 nsecs.There are 100 “triggers” per secondEach triggered event is 1 MByte in sizePhysics data cachePBytes/sec 622 Mbits/secStorageStorageStorageTier2 Centre 1 TIPSTier2 Cen

58、tre 1 TIPSTier2 Centre 1 TIPSAustralian Centre 1 TIPS622 Mbits/sec1 TIPS is approximately 25,000 SpecInt95 equivalentsTier 4Tier 0Tier 1Tier 2Tier 33. Grid Use in Business Enterprise Computing ApplicationTraditional Model Grid-based ModelEmail serverWebserverDatabaseserverAppsserverUpgrade to a new

59、serverto handlemore usersUtilise IT infrastructure effectivelyService Virtualization Layer & Load BalancingOutlineIntroductionUtility Networks, Trends, and Grid ComputingDefining GridsWhat is Grid?, Types of Grid Services Global Grids and ChallengesCharacteristics, Security, resource management, pricing,Grid ArchitectureLayered architectureGrid TechnologiesGlobus from USA and Gridbus from AustraliaGrid ApplicationsE-Science and e-BusinessConclusionConclude with a comparison to t