短发女生穿衣搭配技巧.ppt

Tracking Metadata and Lineage of the Data Processing Chain for Mapping Snow Cover Properties with the NASA MODIS,James Frew1, Thomas H. Painter2, Peter Slaughter1, Jeff Dozier1,1Donald Bren School of Environmental Science and Management, University of California, Santa Barbara 2National Snow and Ice Data Center, University of Colorado, Boulder,女装批发 ,Outline,Motivation Snow mapping product Implications for hydrologic modeling Lineage Capture Wrapping: the ESSW experience Instrumenting, overriding, monitoring: the (ongoing) ES3 experience,MODIS image Sierra Nevada,EOS Terra MODIS 07 March 2004 MOD09 Surface Reflectance 0.555 0.645 0.858,4,Snow-covered area and grain size,Hindu Kush,2003 DOY 070,Colorado Rockies CLPX 13 March 2002,Model structure: MODIS snow-area / albedo,Lineage Capture, Take 1,The ESSW experience,Using Existing Science Applications,No “standard” Earth science computing environment commercial packages (ArcInfo, MATLAB, ) public packages/models (MM5, MODTRAN, ) locally-developed codes arbitrary combinations of Example: SST from AVHRR commercial, standalone programs parameters highly customized for UCSB How do we get these programs to communicate cooperate with ESSW, without rewriting them?,Navigate (Manual/Automatic),Receive,Ingest and Calibrate,Rectify,Sea Surface Temp (SST),SST Maps,Lineage: Current Best Practice,Earth System Science Workbench (ESSW),Producer and consumer issues can both be addressed by a laboratory metaphor Experiment Network of models ingesting / synthesizing data generating products Laboratory Experiment execution environment Computing + storage = accessibility + scalability Lab Notebook Persistent storage that can be queried Keeps track of all experiments Documentation + lineage = accountability,Wrap Your App: Scripts Talk to ESSW,No changes, just additions Wrapper scripts Make program (groups) look like ESSW experiments use Perl API Lab Notebook daemon Accepts API commands Creates XML documents Sends to database ESSW database XML metadata & DTDs Tabular metadata XML search terms Lineage links,Navigate (Manual/Automatic),Receive,Ingest and Calibrate,Rectify,Sea Surface Temp (SST),SST Maps,ESSW Database,Perl API,Lab Notebook daemon,XML + SQL,MySQL,JDBC,Java,Perl,ESSW Metadata management,Lab Notebook daemon verifies XML metadata document Experiment step metadata stored for product lineage tracking Complete metadata document stored in custom database table XML DTD 1:1 database table (n+1)th column is document itself Some metadata values extracted into database tables DTD contains column names and types for some elements Always save all the XML, even if dont know how to “columnize” all of it,Wrapper Example: Input Dataset,Wrapper Example: Output Dataset,Wrapper Example: Process,Wrapper Example: Lineage Links,Process graph reconstructed from ESSW database,ESSW Lessons,Providers are customers ESIPs arent much good unless scientists are happy to put information in them A light touch is the right touch Wrapping is easier for scientists and their programmers to deal with than complete re-engineering Scientists do write scripts, but not necessarily Perl Scripting (gluing stuff together) comes naturally to scientists Scientists dont write DTDs Nobody calls metadata APIs ESSW was automatic, but not automatic enough,Lineage Capture, Take 2,The ES3 experience,ES3 : Earth System Science Server,ESSW+ data lineage tracking,BUB data storage,ROCKS processing clusters,Alexandria Digital Library,Microsoft TerraServer,MODster,OpenDAP,MODIS,Corona,AVHRR,Watershed-scale snow product,Global-scale snow product,From ESSW to ES3: Summary,Perl wrappers “Probulators” Perl API web services + XML messages MySQL XML database(s),From Wrappers to Probulators,Wrappers: Active Lineage + Complete control over what gets recorded Single language/API for all wrapped events Not tied to execution You can even lie about what happened Must explicitly script everything Scripts can drift from reality You can even lie about what happened,From Wrappers to Probulators,Probulators: Passive Lineage + Record what actually happened Not just what you think happened Not what didnt happen Automatic: dont have to write new scripts for everything Different flavors for different environments Cant just do everything in Perl,Probulator patterns,Instrumentation Insert lineage capture instructions directly into science codes e.g. “I just created file foo” Typical implementation: preprocessor/precompiler Overriding Replace standard routines/libraries with lineage-capturing versions e.g. open() snoopy_open() Typical implementation: modify execution environment environment variables configuration files Passive monitoring Trace program execution e.g. “called open() with args foo, bar, ” Typical implementation: straced shell,ES3 Lineage Architecture,probulator1,probulatorn,logger,transmitter,ES3 core,Probulating IDL: Instrumenting the code,;edit pro modscag_cleanse,prefix=prefix,ns=ns,nl=nl HELP, NAMES=“*“, OUTPUT=ES3_ENVIROMENT modscag_cleanse,Probulating IDL: Results, /home/rsi /home/haavar/painter/data/tillsnow.pic ,Probulating bash: Passive Monitoring,cat /etc/passwd | grep haavar | sed -n s/(.*:)2(0-9+).*/2/p 25232 1138336174.480079 open(“/etc/ld.so.cache“, O_RDONLY) = 3 25232 1138336174.480215 open(“/lib/libm.so.6“, O_RDONLY) = 3 25234 1138336178.887267 dup2(3, 255) = 255 25234 1138336178.887912 pipe(3, 4) = 0 25234 1138336178.888257 clone(child_stack=0, , child_tidptr=0xb7f2e708) = 25235 25235 1138336178.889366 dup2(4, 1) = 1 25235 1138336178.889975 pipe(3, 4) = 0 25235 1138336178.890326 clone(child_stack=0, , child_tidptr=0xb7f2e708) = 25236 25235 1138336178.891260 pipe(4, 5) = 0 25235 1138336178.891756 clone(child_stack=0, , child_tidptr=0xb7f2e708) = 25237 25235 1138336178.892753 clone(child_stack=0, , child_tidptr=0xb7f2e708) = 25238 25238 1138336178.894266 dup2(4, 0) = 0 25236 1138336178.894726 dup2(4, 1) = 1 25237 1138336178.894763 dup2(3, 0) = 0 25237 1138336178.895581 dup2(5, 1) = 1 25238 1138336178.897006 execve(“/bin/sed“, “sed“, “-n“, “s/(.*:)2(0-9+).*/2/p“, “HOSTNAME=“, “TERM=xterm-color“, 25236 1138336178.900117 execve(“/bin/cat“, “cat“, “/etc/passwd”, 25237 1138336178.903342 execve(“/bin/grep“, “grep“, “haavar“, ,Probulating bash: Results, same as IDL /etc/passwd /etc/ld.so.cache /etc/passwd haavar ,Now What?,Probulator reports not universally unique Q: How hook separate reports together? A: Logger assigns UUIDs to Data streams Processes Jobs (workflows) Lineage not explicit Q: How publish lineage? A: ES3 Core builds serialized graph,Thanks to:,Current Mike Colee Stephane Maritorena Dominic Metzger Karl Rittger Dave Siegel,Former Anurag Acharya Rajendra Bose Scott Denning Debbie Donahue Jim Duff Calin Duma Erik Fields Jim Gray Steve Miley Jordan Morris Mark Pelletier Pete Peterson Walter Rosenthal Klaus Schauser Hvar Valeur,To Probulate Further : Publications,Bose, R. and Frew, J., 2005. Lineage retrieval for scientific data processing: a survey. ACM Computing Surveys, vol. 37, no. 1, pp. 1-28. do