Globus入门指南.ppt

1、,A Globus PrimerWhat is the Grid and How Do I Use It?,Lee LimingU. of Chicago and Argonne National LaboratoryThe Globus AllianceGRIDS Center/NSF Middleware Initiative,GlobusWORLD 2005,Globus Primer,2,Approach,What is the Grid? How are Grids built and used (today)? What Grid software is available and

2、 what does it do? BREAK (somewhere during part III) How have others succeeded?,I. What is the Grid?,GlobusWORLD 2005,Globus Primer,4,History,In the early 90s, Ian Foster (ANL, U-C) and Carl Kesselman (USC-ISI) enjoyed helping scientists apply distributed computing. Opportunities seemed ripe for the

3、picking. Application of technology always uncovers new and interesting requirements. Science is cool! Big/Innovative science is even cooler!,GlobusWORLD 2005,Globus Primer,5,What Kinds of Applications?,Computation intensive Interactive simulation (climate modeling) Very large-scale simulation and an

4、alysis (galaxy formation, gravity waves, battlefield simulation) Engineering (parameter studies, linked component models) Data intensive Experimental data analysis (high-energy physics) Image and sensor analysis (astronomy, climate study, ecology) Distributed collaboration Online instrumentation (mi

5、croscopes, x-ray devices, etc.) Remote visualization (climate studies, biology) Engineering (large-scale structural testing, chemical engineering) In all cases, the problems were big enough that they required people in several organization to collaborate and share computing resources, data, instrume

6、nts.,GlobusWORLD 2005,Globus Primer,6,What Types of Problems?,Your system administrators cant agree on a uniform authentication system, but you have to allow your users to authenticate once (using a single password) then use services on all systems, with per-user accounting. You need to be able to o

7、ffload work during peak times to systems at other companies, but the volume of work theyll accept changes from day-to-day.,GlobusWORLD 2005,Globus Primer,7,What Types of Problems?,You and your colleagues have 6000 datasets from the past 50 years of studies that you want to start sharing, but no one

8、is willing to submit the data to a centrally-managed storage system or database. You need to run 24 experiments that each use six large-scale physical experimental facilities operating together in real time.,GlobusWORLD 2005,Globus Primer,8,Some Core Problems,Too hard to keep track of authentication

9、 data (ID/password) across institutions Too hard to monitor system and application status across institutions Too many ways to submit jobs Too many ways to store & access files and data Too many ways to keep track of data Too easy to leave “dangling” resources lying around (robustness),GlobusWORLD 2

10、005,Globus Primer,9,Challenging Applications,The applications that Grid technology is aimed at are not easy applications! The reason these things havent been done before is because people believed it was too hard to bother trying. If youre trying to do these things, youd better be prepared for it to

11、 be challenging. Grid technologies are aimed at helping to overcome the challenges. They solve some of the most common problems They encourage standard solutions that make future interoperability easier They were developed as parts of real projects In many cases, they benefit from years of lessons f

12、rom multiple applications Ever-improving documentation, installation, configuration, training,GlobusWORLD 2005,Globus Primer,10,Earth System Grid,Goal: address technical obstacles to the sharing & analysis of high-volume data from advanced earth system models,ESG,GlobusWORLD 2005,Globus Primer,11,Co

13、llaborative Engineering:NEESgrid,U.Nevada Reno,,GlobusWORLD 2005,Globus Primer,12,Enterprise Workload Management: SAP AG R/3,Goal: Lower cost of operation and improve performance of sales and pricing tools. Applied to R/3 products (CRM, SCM) Adapts to changing demand & resources Disc

14、overs and allocates resources as needed to meet specified response time goals,SAP AG R/3 Internet Pricing & Configurator (IPC),GlobusWORLD 2005,Globus Primer,13,Forget Homogeneity!,Trying to force homogeneity on users is futile. Everyone has their own preferences, sometimes even dogma. The Internet

15、provides the model,GlobusWORLD 2005,Globus Primer,14,Evolution of the Grid,Increased functionality, standardization,Time,Globus Toolkit,Open Grid Services Arch,GGF: OGSI, WSRF, (leveraging OASIS, W3C, IETF) Multiple implementations, including Globus Toolkit,Defacto standards GGF: GridFTP, GSI (lever

16、aging IETF),Custom solutions,X.509, LDAP, FTP, ,Web services,App-specific Services,GlobusWORLD 2005,Globus Primer,15,Open Grid Services Architecture,Define a service-oriented architecture the key to effective virtualization to address vital Grid requirements AKA utility, on-demand, system management

17、, collaborative computing, etc. building on Web service standards. extending those standards when needed,GlobusWORLD 2005,Globus Primer,16,WSRF & WS-Notification,Naming and bindings (basis for virtualization) Every resource can be uniquely referenced, and has one or more associated services for inte

18、racting with it Lifecycle (basis for fault resilient state management) Resources created by services following factory pattern Resources destroyed immediately or scheduled Information model (basis for monitoring & discovery) Resource properties associated with resources Operations for querying and s

19、etting this info Asynchronous notification of changes to properties Service Groups (basis for registries & collective svcs) Group membership rules & membership management Base Fault type,GlobusWORLD 2005,Globus Primer,17,Grid and Web Services Convergence,The definition of WSRF means that the Grid an

20、d Web services communities can move forward on a common base.,GlobusWORLD 2005,Globus Primer,18,Theory - Practice,NOTE: As in the Internet, applications can use capabilities at any level of the stack. Abstractions are provided for convenience.,GlobusWORLD 2005,Globus Primer,19,An “Ecosystem” of Grid

21、 Software,Components fit into “niches.” Theres plenty of room for specialized capabilities. Focus on the requirements of your niche. Dont try to “do it all” or compete with obvious winners. Component “evolution” is encouraged. Experimentation is obligatory. “Failure to thrive” isnt a bad thing in th

22、e big picture. Acceptability (“usefulness”) is the key requirement. Dont protect things unfairly. (A true ecosystem has no internal boundaries.) Good solutions usually “bubble up” from below, as opposed to “falling down” from on high. Cooperative relationships are allowed! Symbiotic relationships ar

23、e ok and may be stronger than a “going it alone” strategy.,GlobusWORLD 2005,Globus Primer,20,A Healthy Ecosystem,Not a monoculture , or Cambrian explosion, but a web of components,II. How are Grids Built and Used (Today)?,GlobusWORLD 2005,Globus Primer,22,Methodology,Building a Grid system or applic

24、ation is currently an exercise in software integration. Define user requirements Derive system requirements or features Survey existing components Identify useful components Develop components to fit into the gaps Integrate the system Deploy and test the system Maintain the system during its operati

25、on This should be done iteratively, with many loops and eddies in the flow.,GlobusWORLD 2005,Globus Primer,23,Software, Standards,A Problem-Driven, Collaborative R&D Methodology,Design,Deploy,Build,Apply,Analyze,Apply,Apply,Deploy,Apply,Computer Science,Infra- structure,Discipline Advances,GlobusWOR

26、LD 2005,Globus Primer,24,Who Is the Grid For?,Any Grid (distributed/collaborative) application or system will involve several “classes” of people. “End users” (e.g., Scientists, Engineers, Customers) Application/Product Developers System Administrators System Architects and Integrators Each user cla

27、ss has unique skills and unique requirements. The user class whose needs are met varies from tool to tool (even within the Globus Toolkit).,GlobusWORLD 2005,Globus Primer,25,What End Users Need,Secure, reliable, on-demand access to data, software, people, and other resources (ideally all via a Web B

28、rowser!),GlobusWORLD 2005,Globus Primer,26,How it Really Happens,WebBrowser,ComputeServer,DataCatalog,DataViewerTool,Certificate authority,ChatTool,CredentialRepository,WebPortal,ComputeServer,Resources implement standard access & management interfaces,Collective services aggregate &/or virtualize r

29、esources,Users work with client applications,Application services organize VOs & enable access to other services,Database service,Database service,Database service,SimulationTool,Camera,Camera,TelepresenceMonitor,RegistrationService,GlobusWORLD 2005,Globus Primer,27,How it Really Happens,Implementat

30、ions are provided by a mix of Application-specific code “Off the shelf” tools and services Tools and services from the Globus Toolkit Tools and services from the Grid community (compatible with GT) Glued together by Application development System integration,GlobusWORLD 2005,Globus Primer,28,How it

31、Really Happens(without the Grid),WebBrowser,ComputeServer,DataCatalog,DataViewerTool,Certificate authority,ChatTool,CredentialRepository,WebPortal,ComputeServer,Resources implement standard access & management interfaces,Collective services aggregate &/or virtualize resources,Users work with client

32、applications,Application services organize VOs & enable access to other services,Database service,Database service,Database service,SimulationTool,Camera,Camera,TelepresenceMonitor,RegistrationService,A,B,C,D,E,GlobusWORLD 2005,Globus Primer,29,How it Really Happens(with the Grid),WebBrowser,Compute

33、Server,GlobusMCS/RLS,DataViewerTool,CertificateAuthority,CHEF ChatTeamlet,MyProxy,CHEF,ComputeServer,Resources implement standard access & management interfaces,Collective services aggregate &/or virtualize resources,Users work with client applications,Application services organize VOs & enable acce

34、ss to other services,Database service,Database service,Database service,SimulationTool,Camera,Camera,TelepresenceMonitor,Globus IndexService,GlobusGRAM,GlobusGRAM,GlobusDAI,GlobusDAI,GlobusDAI,GlobusWORLD 2005,Globus Primer,30,Planning Tasks,Identify project/application goals Identify specific funct

35、ional requirements Identify/develop social policies, procedures, processes Identify components to meet requirements Form an integration plan and identify functional gaps Implement the product Deploy the product Provide O&M for the product, assess status and iterate,GlobusWORLD 2005,Globus Primer,31,

36、Application “Vision”,Never underestimate the amount of work that can be wasted for lack of a specific target. Desired end user capabilities (accomplishments?) leads to: specific end user goals, milestones leads to: specific system requirements leads to: specific system features leads to: specific sy

37、stem components A specific target application/system will focus the work dramatically. For example: Enable engineers to run many more simulation runs. Enable realtime remote observation of MRI scanning. Provide broad, easy-to-use access to satellite data. Encourage scientists to share/use experiment

38、al data.,GlobusWORLD 2005,Globus Primer,32,Functional Requirements,What capabilities must the system or application provide? “What?” Not “How?” A common mistake is to select components before defining what is really needed. Requirements should clearly relate to project goals. Examples: Experiment pa

39、rticipants must be able to communicate in realtime with each other. The system must be able to manage and carry out a large number of inter-related tasks efficiently. Detailed records of how data was derived must be kept and made available to data users. Data must be automatically replicated among s

40、ervers to optimize proximity based on past use. Access control must be employed across the system, with end-user “single sign-on.”,GlobusWORLD 2005,Globus Primer,33,Social Policies/Procedures,How will people use the system? Who will set up access control? Who creates the data? How will computational

41、 resources be added to the system? How will simulation capabilities be used? What will accounting data be used for? Not all problems are solved by technology! Understanding how the system will be used is important for narrowing the requirements.,GlobusWORLD 2005,Globus Primer,34,Architecture,System

42、architecture should be coherent, modular, flexible, simple, and mandatory. The earlier you produce and share a project-wide architecture document, the more it will be used. The design will be iterated on, so get it out early! The cost of deviation can be quite painful. Duplication of effort Incompat

43、ible components Complicated/unworkable deployment challenges A bad user experience Working by Consensus does not work in a distributed development activity. A strong software manager should lead the charge and ensure that all teams are working in cohesion.,GlobusWORLD 2005,Globus Primer,35,Architect

44、ure,Once you have some decent requirements and some understanding of use cases Draw the system design. Describe how the design will meet the needs of typical use cases. Consider deployment and M&O requirements for the design. Get feedback! You will start getting a sense of what components will be ne

45、eded.,GlobusWORLD 2005,Globus Primer,36,Select Components,Within the system design, components will have functional requirements, too. Capabilities (“features”) Interfaces (protocols, APIs, schema) Performance/scalability metrics Ideally, much of it already exists. Leverage whats already out there (

46、Web, Grid, fabric technologies, off-the-shelf products, etc.). Decompose into smaller bits if necessary. If too much is unique to this application, youre probably doing something wrong. If a candidate component is almost-but not quite-perfect, it can probably be extended (or used in conjunction with

47、 something else) to meet requirements.,GlobusWORLD 2005,Globus Primer,37,Application Development,Phased “top-down” development Focus on satisfying individual project goals or requirements in turn, or Focus on widening deployment in turn. Danger of “muddying” the architecture (inefficiencies creep in

48、, especially regarding reusability). “Bottom-up” development Focus first on components, then move to “system integration”. Danger of missing the “big picture” (missing unstated requirements).,GlobusWORLD 2005,Globus Primer,38,Continuous Deployment,Involve “real users” as early as possible Youll lear

49、n a lot and be able to “course correct” You will establish a set of happy users to help down the road Pick early adopters carefully. Aggressive users, technologically skilled, representative of the target user base. Set expectations carefully. Be wary of over-investment. Deployment is a significant

50、chunk of your effort. Separate team? Make sure its linked to the development activity. Demonstrate results early and often, and work with new users to get an “ownership” of the code and features,GlobusWORLD 2005,Globus Primer,39,Deployment,Involve “real users” as early as possible. Youll learn a lot

51、 and be able to “course correct.” Youll establish “happy users” to help in later stages. Pick early adopters carefully. Aggressive users, technologically skilled, representative of the target user base. Set expectations carefully. Be wary of overinvestment. Deployment is a significant chunk of your

52、effort. Separate team? Make sure its linked to the development activity.,GlobusWORLD 2005,Globus Primer,40,Evolution & Adaptation,Cost/benefit of “improving” system components has to be considered carefully. What is the benefit offered by the changes? What else changes from the users perspective? Ho

53、w many people (users, administrators, trainers, tech support, ) would be affected? How much “deployment and use” investment would be lost? (Documentation, training, redeployment, integration, app development, data conversion, etc.) Most costs increase as time passes, assuming youve been engaging the

54、 community successfully.,GlobusWORLD 2005,Globus Primer,41,Expertise,The following types of skills are needed. System Administration System Integration Application Development Planning/Management You need less of these with existing Grid tools than without them. Existing services can be re-used. API

55、s make development of new services easier. In most projects, individuals will combine skills above. (You dont need one person for each.),GlobusWORLD 2005,Globus Primer,42,Prerequisite Hardware/Software,Most Grids make heavy use of Unix/Linux Open Source and easily extensible Decent price/performance

56、/reliability/native security More familiar to original Grid community Windows .NET is good for OGSI/WSRF-based work Yet another benefit of being Web service standards compatible Hardware depends on application goals Clusters for compute-intensive apps, highly-parallel work (including data striping)

57、and databases Mass storage for data-intensive apps Networking for communication-intensive apps (data transfer, videoconferencing, remote visualization, etc.),GlobusWORLD 2005,Globus Primer,43,Important Planning Considerations,All Grid technology is evolving rapidly. Web services standards Grid inter

58、faces Grid implementations Grid hosting services (ASP, SSP, etc.) Community is important! Best practices (GGF, OASIS, etc.) Open source (Linux, Axis, Globus, etc.) Application of community standards is vital. Increases leverage Mitigates (a bit) effects of rapid evolution Paves the way for future in

59、tegration/partnership,III. What Grid Software is Available and What Does It Do?,GlobusWORLD 2005,Globus Primer,45,An “Ecosystem” of Grid Software,The role of the Globus Toolkit A tour of todays ecosystem Security Monitoring and Discovery Computation/Processing Data Collaboration Packaging/Distribution,GlobusWORLD 2005,Globus Primer,46,What Is the Globus Toolkit?,The Globus Toolkit is a collection of solutions to problems that frequently come up when trying to build collaborative distributed applications. Heterogeneity To date (v1.0 - v4.0), the Toolkit has focused on simplifyi