地球信息科学导论2-RS2

1、地球信息科学导论Introduction to geo-information scienceSpring 20162. Metadata, catalogue, catalogue federation,白玉琪 清华大学地球系统科学研究中心 ,数据 采集,数据 管理,数据 展示,数据 分析,数据需求者,数据提供者,数据生命周期,Starting from NASA Reverb,Search criteria (1),Search criteria (2),Search criteria (3),Search criteria (4),Search results,Question?,How

2、 to effectively find matched images?,Effective discovery of geospatial data: a geospatial catalogues perspective,Contents,Geospatial data discovery problems Geospatial data discovery systems System architectures Referenced metadata standards Referenced Catalogue Service standard Geospatial Catalogue

3、 Federation Case study: GMU CSISS CFS product Main challenges Proposed federation strategies Product system Discussion GMU CSISS CSW/CFS Applications Summary,Background,Large volume of geospatial data has been accumulated over the last several decades through mapping, survey and observation Petabyte

4、 level NASA EOSDIS project is expected to archive one petabyte per year of raw data that are distributely managed in data centers. On November 20, 2003, the NASA Land Processes Distributed Active Archive Center (LP DAAC) data archive holdings crossed the one petabyte threshold in volume*. 1 petabyte

5、 = 1,000,000,000,000,000 bytes = 8,000,000 Second * 1 Gb/s ( 92.59 days) = 80,000,000 Seconds * 100 Mb/s (925.9 days),*/petabyte.asp,Problems and Questions,Problems Large volume of geospatial data has to be maintained in few data centers, while these data are highly needed in e

6、very piece of research directed by research staffs, professors and students in every college, university and government agency.,geospatial data,Questions How to help users to evaluate the fitness for use of a particular data set, among hundreds of collections and millions of granules, for their spec

7、ific decision or assessment?,End user,Geospatial Data Discovery Mechanism,Walk around Organizing textual information about the identification, the extent, the quality, the spatial and temporal schema, spatial reference, and distribution of every piece of data set Metadata (data about data),geospatia

8、l data,geospatial metadata,discovery interface,End user,Providing catalogue discovery interface against these metadata information for end users,Enabling direct data download or customization through online software modules, or “services”,Geospatial Metadata,The metadata required for effective data

9、management varies with the type of data and context of use. Standards + Profiles,geospatial data,geospatial metadata,discovery interface,End user,Standards: ISO ISO 15836:2003 Dublin Core metadata element set Stage: 60.60 (2003-11-26) ISO 19115:2003 Geographic information - Metadata Stage: 60.60 (20

10、03-05-08) ISO 19115:2003/Cor 1:2006 Stage: 60.60 (2006-07-05) ISO 19115-2 Geographic information - Metadata - Part 2: Extensions for imagery and gridded data Under development Stage: 40.00 (2007-10-25) ISO 19119:2005 Geographic information - Services Stage: 60.60 (2005-02-10) ISO 19139:2007 Geograph

11、ic information - Metadata - XML schema implementation Stage: 60.60 (2007-04-17),Geospatial Metadata (Cont.),The metadata required for effective data management varies with the type of data and context of use. Standards + Profiles,geospatial data,geospatial metadata,discovery interface,End user,Stand

12、ards: US FGDC-STD-001-1998 Content Standard for Digital Geospatial Metadata FGDC-STD-012-2002 Content Standard for Digital Geospatial Metadata: Extensions for Remote Sensing Metadata NASA ECS Science Metadata,Geospatial Metadata Discovery Interface,geospatial data,geospatial metadata,discovery inter

13、face,End user,The discovery interface varies with the type/structure of underlying metadata and context of use.,Fromthe users point of view: Simple web page navigation with no search functionality E.g. THREDDS,Geospatial Metadata Discovery Interface (Cont.),The discovery interface varies with the ty

14、pe/structure of underlying metadata and context of use.,geospatial data,geospatial metadata,discovery interface,End user,Fromthe users point of view: Simple web page navigation with no search functionality E.g. THREDDS Web page navigation with limited search functionalities E.g. NASA GCMD,Geospatial

15、 Metadata Discovery Interface (Cont.),The discovery interface varies with the type/structure of underlying metadata and context of use.,geospatial data,geospatial metadata,discovery interface,End user,Fromthe users point of view: Simple web page navigation with no search functionality E.g. THREDDS W

16、eb page navigation with limited search functionalities E.g. NASA GCMD Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG),Geospatial Metadata Discovery Interface (Cont.),The discovery interface varies with the type/structure of underlying metadata

17、and context of use.,geospatial data,geospatial metadata,discovery interface,End user,Fromthe users point of view: Simple web page navigation with no search functionality E.g. THREDDS Web page navigation with limited search functionalities E.g. NASA GCMD Web-based GUI with enhanced search functionali

18、ties, no public API interface E.g. EOS Data Gateway (EDG),LP DAAC,GES DISC,Geospatial Metadata Discovery Interface (Cont.),The discovery interface varies with the type/structure of underlying metadata and context of use.,geospatial data,geospatial metadata,discovery interface,End user,Fromthe users

19、point of view: Simple web page navigation with no search functionality E.g. THREDDS Web page navigation with limited search functionalities E.g. NASA GCMD Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG) Web-based GUI with enhanced search functi

20、onalities, with proprietary API interface E.g. NASA ECHO IIMSAQL Query Language,ECHO Service Core,GES DISC,LP DAAC,GMU CSISS ECHO OGC Wrapper,geospatial data,geospatial metadata,discovery interface,End user,15 Terabytes Images,GMU CSISS OGC Catalogue Service Core,ebRIM Wrapper,ISO Wrapper,OGC Core,D

21、ata Download,GeoBrain Online Analysis System (GeOnAS),Geospatial Metadata Discovery Interface (Cont.),The discovery interface varies with the type/structure of underlying metadata and context of use.,Fromthe users point of view: Simple web page navigation with no search functionality E.g. THREDDS We

22、b page navigation with limited search functionalities E.g. NASA GCMD Web-based GUI with enhanced search functionalities, no public API interface E.g. EOS Data Gateway (EDG) Web-based GUI with enhanced search functionalities, with proprietary API interface E.g. NASA ECHO IIMSAQL Query Language Web-ba

23、sed GUI with enhanced search functionalities, with open API interface E.g. GMU CSISS/LAITS CSW,GMU CSISS/LAITS CSW,- Designed and Developed from Aug. 2003- Support OGC CSW 2.0.1 and 2.0.2,Geospatial Catalogue Service Standard,OGC Catalogue Service is the only available standard specifies the interfa

24、ces between clients and catalogue services through the presentation of abstract and implementation-specific models. Catalogue Service and its clients OGCs perspective: Catalogue Service supports the ability to publish and search collections of descriptive information (metadata) for data, services, a

25、nd related information objects. Metadata in catalogues represent resource characteristics that can be queried and presented for evaluation and further processing by both humans and software. Catalogue services are required to support the discovery and binding to registered information resources with

26、in an information community.,geospatial data,geospatial metadata,discovery interface,End user,Catalogue Service,Catalogue Service Client,Geospatial Catalogue Service Standard (Cont.),/standards/cat,Geospatial Catalogue System,geospatial data,geospatial metadata,discovery

27、interface,End user,New Problems and Questions,geospatial data,geospatial metadata,discovery interface,End user,New Problems and Questions,geospatial data,geospatial metadata,discovery interface,End user,New Problems and Questions,geospatial data,geospatial metadata,discovery interface,End user,New P

28、roblems and Questions,geospatial data,geospatial metadata,discovery interface,End user,New Problems and Questions,Different agencies have developed their own geospatial catalogues to facilitate discovery, access, and sharing of large volumes of geospatial data, either observed satellite images or si

29、mulation data. These geospatial catalogues are becoming accessible online through their query interfaces. For scientists who conduct multi-disciplinary research, they may need to search multiple catalogues in order to find the data they need. Such work is very time-consuming and tedious, especially

30、when the catalogues may use different metadata models and catalog interface protocols. It is very desirable if those catalogues can be integrated into a catalogue federation, which will present a well-known metadata model and interface protocol to users and hide the complexity and diversity of the a

31、ffiliated catalogues behind the interface. With the federation, users only need to work with the federated catalogue to find the data they need instead of working with individual catalogues individually. Catalogue federation service - integrating multiple legacy catalogues to facilitate distributed

32、and integrated data discovery.,Federation Context,geospatial data,geospatial metadata,discovery interface,End user,Catalogue Federation,Federation Case Study GMU CSISS CFS System,Community Catalogues,End user,GMU CSISS Catalogue Federation Service,NASA ECHO,GMU CSISS OGC CSW,DOE Earth System Grid Si

33、mulation Data Catalogue,discovery interface,GMU GUI,Third Party System,Federation Case Study GMU CSISS CFS System (Cont.),We analyzed each catalogue in the following aspects: Metadata Conceptual Model Query Language Communication Protocol,Federation Case Study GMU CSISS CFS System (Cont.),Challenges

34、 in Federating NASA ECHO, GMU CSW, and ESG Catalogues are: 1. Protocol Adaptation GMU CSW and the ESG catalogue support HTTP protocol (GET/POST) binding, while NASA ECHO uses SOAP to maintain the connection with the clients. The federation server should use the correct protocol when communicating wi

35、th each Catalogue service. The protocol the clients may use to talk to the federation server itself is another concern. After all protocols have been defined and identified, the federation server should support protocol adaptation internally.,Federation Case Study GMU CSISS CFS System (Cont.),Challe

36、nges in Federating NASA ECHO, GMU CSW, and ESG Catalogues are: 2. Query Dispatching The federation server is responsible for dispatching a query to the affiliated catalogue services. A dispatching model should be defined to deal with the following issues: Transparency: Whether the federation user is

37、 aware of these affiliated catalogue services and whether users can define which catalogue services are of interest in their queries. Sequence: Whether the federation server dispatches the users queries to these affiliated catalogue services in a predefined sequence, whether this sequence can be cha

38、nged in runtime, and whether the federation users can define this sequence in their queries.,Federation Case Study GMU CSISS CFS System (Cont.),Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues are: 3. Query Translation: The translation of queries is another major issue. The federation

39、 has to deal with the following problems: Metadata Query Objects: The metadata objects queried against using one set of query criteria may not have counterparts in another schema. For example, the federation service cannot fulfill queries for objects defined in GMU CSW and NASA ECHO for those simula

40、tion-specific metadata objects referenced only in the ESG catalogue schema. Another issue is that the same registry object has different names, in different schemes, e.g., Granule in NASA ECHO versus DataGranule in GMU CSW. Query Format: Both GMU CSW and the ESG Catalogue accept queries in OGC Filte

41、r format, while ECHO only accepts IIMSAQL format. The federation server needs to transform an individual query into the different proprietary formats. The spatial query criterion and temporal query criterion are expressed differently in the NASA ECHO granule query payload and the GMU CSW granule que

42、ry payload. Query Language Functionality: Some complex query predicates in one query language cannot be identically expressed in another one. For example, the OGC Filter specification supports nested Boolean queries. Such queries can be supported at best with difficulty on ECHO IIMSAQL, and some can

43、not be supported at all.,Federation Case Study GMU CSISS CFS System (Cont.),Challenges in Federating NASA ECHO, GMU CSW, and ESG Catalogues are: 4. Results Integration: Catalogue query results from multiple Catalogue Services may need to be integrated before being sent back to users. As these metada

44、ta results may not use the same schema, the rules the federation server uses to re-organize metadata information while keeping the original content should be well designed. Furthermore, whether the clients can define the format of the query result of interest and, if so, how, also needs to be addres

45、sed.,Federation Case Study GMU CSISS CFS System (Cont.),We proposed the following federation strategies: 1. Protocol Adaptation As this federation is supposed to provide a single access point to multiple, autonomous information sources, it may follow the mediator-wrapper architecture, where the fede

46、ration works as a mediator, and wrappers may be deployed for communicating with specific catalogue services if protocol adaptation is needed. 2. Query Dispatching 1. Opaque: In this scenario, the federation service fully controls the distributed query process, with the clients having no awareness of

47、 the affiliated Catalogue Services. 2. Translucent: The federation service may expose the affiliated Catalogue Services to the users, but the users can define neither which Catalogue Services their query can be forwarded to nor the sequence of queries. 3. Transparent: The federation service may expo

48、se the affiliated Catalogue Service to the users, and the users can define those Catalogue Services of interest and the sequence in which their queries can be dispatched.,Federation Case Study GMU CSISS CFS System (Cont.),Proposed federation strategies: 3. Query Translation Query Translation in fede

49、ration has two aspects: semantic and syntactic. A federation usually maintains a global schema that is exposed to end-users. Metadata attribute terms in user queries always follow this global schema. Before being dispatched to an underlying affiliated catalogue service, they should be transformed ap

50、propriately. This transformation logically involves four layers: metadata term, query criterion, query criteria, and query payload, as shown in the following picture.,Federation Case Study GMU CSISS CFS System (Cont.),Proposed Federation Strategy 4. Query Result Integration A federation service need

51、s to integrate query results from multiple underlying Catalogue Services before sending them back to the clients. It may choose to implement one of three kinds of integration mechanisms. Opaque: In this case, the federation service defines, maintains and advertises a unique information model. Each q

52、uery result from affiliated Catalogue Services should, if necessary, be transformed to this information model. The original metadata information can be kept in the final transformed query results. Translucent: The federation service does not maintain a complete, unique information model but defines

53、a common subset of metadata objects that are supported by all the affiliated Catalogue Services, such as name, and spatial and temporal range. The federation service transforms only this part of the metadata information, while the remaining embedded original metadata information remains unchanged in

54、 the final response. Transparent: The federation service has no role in metadata integration. All the query results from affiliated Catalogue Services are simply grouped together, keeping the original metadata formats. In this scenario, the users are supposed to analyze each result fetched from fede

55、ration service, since the results may not all conform to the same schema even though grouped together in one response.,Federation Case Study GMU CSISS CFS System (Cont.),Proposed Federation Strategy 4. Query Result Integration A federation service needs to integrate query results from multiple under

56、lying Catalogue Services before sending them back to the clients. It may choose to implement one of three kinds of integration mechanisms. Opaque: In this case, the federation service defines, maintains and advertises a unique information model. Each query result from affiliated Catalogue Services s

57、hould, if necessary, be transformed to this information model. The original metadata information can be kept in the final transformed query results. Translucent: The federation service does not maintain a complete, unique information model but defines a common subset of metadata objects that are sup

58、ported by all the affiliated Catalogue Services, such as name, and spatial and temporal range. The federation service transforms only this part of the metadata information, while the remaining embedded original metadata information remains unchanged in the final response. Transparent: The federation

59、 service has no role in metadata integration. All the query results from affiliated Catalogue Services are simply grouped together, keeping the original metadata formats. In this scenario, the users are supposed to analyze each result fetched from federation service, since the results may not all conform to the same schema even though grouped together in one response.,Federation Case Study GMU CSISS CFS System (Cont.),Federation Syste