数据结构本科教学资料-hdf数据结构的研究

1、hdf数据结构研究1. hdf概念hdf, hierarchical data format,分层数据格式，美国国家高级计算应用中心ncsa (national center for supercomputing applications)为了满足各领域研究需求而研制 的一种能高效存储和分发科学数据的新型数据格式，是一种分层式数据管理结构，被地球观 测系统数据和信息系统核心系统所选用作为标准数据格式。它的表现形式是一种多目标的文 件格式，目的是为了在分布式环境中共享科学数据。ndf文件格式的优势在于：可移植性强(独立于操作平台)；属于超文本文件；可以存 储并处理大数据量；一个文件集可以管理多种

2、类型的数据结构；具有可扩展性。由于hdf的 诸多优点，这种格式已经被广泛用于目前国外各种卫星传感器的标准数据格式。在影像数据 库多源数据管理屮，hdf格式发挥了很好的作用，利用hdf数据结构建立远程图彖工程，并与 数据库进行交互；可以进行远程图像处理；远程影像解译，统计分析；影像运算、信息挖掘、 影像分类；综合处理影像、矢量、高程数据，三维可视显示等。2. hdf数据结构hdf文件有六种主要数据类型，如图：1栅格图像raster image：数据模式提供一种灵活方式存储、描述栅格图像数据，包括 8bit栅格图像。2. 调色板palette：也叫作彩色查对表，它提供图像的色谱。3. 科学数据集s

3、cientific data set：用来存储和描述多维科学数据陈列。4. hdf注释annotation：是文字串，用来描述hdf文件或hdf数据目标。5. vdata：是一个框架，用于存储和描述数据表。6. vgroup：是用来把相关数据目标联系起来。一个vgroup可以含有其它vgroup,以及数据目标。任一个hdf目标均可以包扌舌进某个vgroup屮。palctlcvdata (table)vgroup(group of hdf data structures)而在hdf5中，建立一个新的hdf文件后，会有5种数据格式显示:hdf4中只有前三种，但nasa下來的数据再hdfview 里

4、面显示的是版本4.o group駐 dataset亀 irnayetable芬 datatype 邂 link3. 对 于 nasa 的 merra300.prod.assim.inst6_3d_ana_nv.2010o531.hdf进行的格式研究。其中有基于sd (science data)和gd (vgroup id)两种方式对hdf文件进行处理,例 如下图，eosgrid就是一个vgroup,而ps就是一个sd的datafieldo利用sd方式进行读写1) sd方式读収的变量，是整个hdf文件的变量，没有层次结构的显示，下面是输出的和hdfview比对的结果。function hdf_s

5、d_vardir, hdfidvarnames =''hdf_sd_fileinfo, hdfid, nvars, ngatts if (nvars gt 0) then beginvarnames = strarr(nvars)for index 二 ol, nvars - il do begin varid = hdf_sd_sclcct (hdfid, index) hdf_sd_getinfo, varid, name=name hdf sd endaccess, varid varnnmes index = nameendforendifreturn, varname

6、sendidl> pp=hdf_sd_vardir(hdfid)idl> print, ppft merra300ins|6_3cf egsgroi|科丹理 delpa毎03送:heigws毬 tutw qj grid mnbjtew>tnnrieosorid fdrri&rrsion) ：3 ydm eosgrid :dmemion) 昌| h?(dimemwg(dimens?cn>ps delp t u v qv 03 xdim:eosgrid ydim:eosgrid height:eosgrid time:eosgrid xdim ydimheight

7、time2) sd方式读取的属性，当变量输入为空吋，显示的是整个hdf文件的属性。下面是输出 的和hdfview比对的结果。function hdf_sd_attdir, hdf 11), varnameif (n_elements(hdfid) eq 0) then $message, ' hdfid is undefincd'attnames =if (varname eq ' ') then beginhdf_sd_fileinfo, hdfid, nvars, nattsendif else beginindex = hdf_sd_nametoindex

8、(hdfid, varname)varicl = hdf_sd_select (hdf id, index)hdf_sd getinfo, varid, natts=nattsendelseif (natts gt 0) then beginattnames 二 strarr(natts)for index = 0l, natts-1 do beginif (varname eq '') then beginhdf_sd_attrinfo, hdfid, index,name=nameendif else beginhdf_sd_attrinfo, varid, index,n

9、ame=nameendelseattnamesindex = nameendforendifif (varname ne " ) then hdf_sd_endaccess, varidreturn, attnamesendidl> kk二hdf_sd_attdir(hdfid,'')idl> print, kkhdfeosversion structmetadata0 missingvalue conventions title history institutionsource references commont contact ks nstep p

10、top pint ak bk coremetadata 0archivedmetadata. 0i general j attributesnumber of attributes = 19addnamevaluetypearray sizehdfeosversio nhdfeos v2.148-bit character12structmetadata.ogroup=swathstructure.8-bit character32000missing value9.9999999e1432-bit floating-point1 |conven tionscf-1.08-bit charac

11、ter6titlemerra reanalysis. geo8-bit character28historyfile written by cfio8-bit character20 |institutio nglobal modeling and ass.8-bit character94sourceglobal modeling and ass.8-bit character54|refere nceshttp:/gmao.gsfc. nasa.go.8-bit character41commentgeos-5.2.08-bit character10contacthttp:/gmao.g

12、sfc. nasa.gow8-bit character26ks4032-bit integer1 |nstep1576032-bit integer1 |ptop1.032-bit floating-point1pint15039.30332-bit floating-point1 |ak1.0,2.0000002,3.27000.32-bit floating-point73bk0.0. 0.0. 0.0, 0.0, 0.0, 0.0.32-bit floating-point73coremetadata.ogroup = inventorym.8-bit character5859arc

13、hivedmetadata.ogroup = archivedme.8-bit character5733) sd的一些概念(即上例打开文件所用的hdfid)科学数据集即sd (science dataset)的td号和已在hdf文件结构屮定义的地球物理参 数名的关系，然后读取数据i1df_sd_getdata到内存中，关闭数据文件hdf_sd_endaccess, hdf_endo每打开一个hdf文件将会分配个sdjd,应该是相当于临时的内存号之类的。(每次的id 值不一样)idl> hdf id=hdf_sd_start (' merra300. prod, assim. i

14、ns16_3d_ana_nv. 20100531. hdf')% loaded dlm: hdf.idl> print, hdfid393216idl> hdfid=hdf_sd_start (，merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf')idl> print, hdfid17170433idl> hdfid二hdf sd start mltrra300. prod, assim. inst6 3d ana nv. 20100531. hdf')idl> print, hdf

15、id339476504) sd方式将已有hdf文件的数据写入单个vgroup的hdf文件。以读取 merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf 的 eosgrid (vgroup)下的 ps dataficld 为例：fid=e0s_gd_0pen(，merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf', /read) grid二 eos_gd_attach(fid,' e0sgrid，)r7 = eos_gd_fieldinfo(grid, * ps，, ra

16、nk, dims, numbertype, dimlist)r6 二 eos_gd_readfield(grid, ps，, data) k二dims pp=fltarr(dims) pp=data hdfid=hdf_sd_start c 20100531. hdf', /create) varid=hdf_sd_create(hdfid,' psd', k, /dfnt_fl0at32) dimid=hdf_sd dimgetid(varid, 0)hdf sd dimset, di mid, name 二'xdim' dimid=hdf_sd_di

17、mgetid(varid,1) hdf_sd_dimset, dimid, name=，ydim" dimid=hdf_sd_dimgetid(varid, 2) hdf sd dimset, dimid, name=，time， hdf_sd_adddata, varid, datahdf_sd_attrset, varid,' qym",' idl examples' hdf_sd_attrset, hdfid,' creation_data，, systime() hdf_sd_endaccess, varidhdf sd end,hd

18、fid此程序是将ps中的数据写入到新建的20100531. hdf文件的psd dataficld中,在hdfvicw中的效果:愆 tableview ps /eosgrid/data fields/ cadocuments and settingsxadrr "me |沏|i 4 i吩|聊|table圃炸o°4 gtableview psd 厂 cadocuments and settingsadministratort20100531 t0123401234067433.3767433.3767433.3767433.3767433.37067433 3767433 3

19、767433 3767433.3767433.37166409.37664253766441 3766473.3766489.37166409.37664253766441.3766473.3766489.37265241.36765273.36765289 36765321.36765337.367265241.36765273 36765289 36765321.36765337.367364441 36764441 36764441 36764457 36764457.367364441.36764441.36764441.36764457.36764457.367464393.3676

20、43933676439336764393.36764409.367464393.36764393 36764393.36764393.36764409.367564841.36764857 36764905.36764937.36765001.367564841.36764857 36764905.36764937.36765001.367665369.36765401.3676543336765497.36765577.37665369.36765401.36765433.36765497.36765577.37766521.3766569.3766601.3766681.3766777.3

21、7766521.3766569.3766601.3766681.3766777.37868297.3768505.3768665.3768825.3768905.37868297.37685053768665.3768825.3768905.37969257.376986537706653772329.3773773.37969257.37698653770665.3772329.3773773.371073073.3773097.3774313.3776931.3779853.371073073.3773097.3774313.3776931.3779853.371190393.378968

22、937895293790985.3792521.371190393.3789689 3789529 3790985.3792521.371298681.3798681 3798681 3798681.3798713.371298681.3798681 3798681.3798681.3798713.371398713.3798713 3798745.3798777.3798777.371398713.3798713 3798745.3798777.3798777.371498777 3798777 3798777 3798809 3798809.371498777.3798777.379877

23、7.3798809.3798809.371598745.3798777.3798777.3798809.3798809.371598745.3798777.3798777.3798809.3798809.371698713.3798745.3798745.3798745.3798745.371698713.3798745 3798745.3798745.3798745.371798649.379864937986493798649.3798681.371798649.3798649.3798649.3798649.3798681.371898553.3798553.3798553.379855

24、3.3798553.371898553.3798553.3798553.3798553.3798553.371998457.3798457.3798457.3798457.3798457.371998457.37984573798457.3798457.3798457.372098393.3798393.3798361.3798361.3798361.372098393.3798393.3798361.3798361.3798361.372198329.37983293798329 3798297.3798297.372198329.37983293798329.3798297.3798297

25、.372298297.37982973798297 3798265.3798265.372298297 3798297 3798297 3798265.3798265.372398297.3798297 3798265 3798265.3798233.372398297.37982973798265.3798265.3798233.372498265.3798233.3798233 3798201.3798201.372498265.3798233.3798233.3798201.3798201.372598233 3798233 3798201 3798201 3798201.3725982

26、33.37982333798201.3798201.3798201.372698265.379823337982333798233.3798201.372698265.3798233.3798233.3798233.3798201.372798265.3798265 37982333798233.3798201.372798265.3798265.3798233.3798233.3798201.372898297.3798265.37982333798233.3798201.372898297.3798265.3798233.3798233.3798201.372998265.3798265.

27、3798233.3798201.3798169.372998265.3798265.3798233.3798201.3798169.373098265.3798233.3798201.3798169.3798137.373098265.37982333798201.3798169.3798137.37讪 imageview - ps - /eosorid/data fields/ c:documents and setting$administratormmerra300.prod.as$im ins16_3d_ana.timage炸10厂4r-注意：1.在hdfview中口己添加datafi

28、eld在idl程序中识别不了。 2. hdfview屮不能删去元素。q2owouva 啲 se.pcudooevalr山左国日a633s 蝴r-ji151關左isooe0础sp o)匚 iq ®a以图形方式打开:qeosceeoa利用gd方式进行读写1 )多个vgroup(rtl gridid标识)打开方法的探究1. 读取vgroup相关信息a. 相关信息：result = eos_gd_gridinfo(g刃d/d, xdimsize, ydunsize, upleft, lowrig/it) this function returns the number of rows, co

29、lumns and the location, in meters, of the upper left and lower right corners of the grid image.b. vgroup 数 fl: result = eos_gd_inqgrid( filename, gridlist , length=variable)c. 直接从 filename 和 gridname 读取相关信息：result = eos_gd_query( filename, gridnanie, info)2. 读取某个vgroup下某个属性的相关东西a. 值：result = eos_gd_

30、readattr(ghd/£>, attmatne, dcitbuf)值将 datbuf 中，returnssucceed(o)讦 successful and fail(-l) otherwise.b .相关信息：result = eos_gd_attrinfo(g/7d/d, attrname, numbertype, count) attribute name (string). a named variable that will contain the number type (long) of an attribute. returns succeed (0) if

31、 successful and fail (-1) otherwise.c.属性数目：result = eos_gd_inqattrs( gridld, attrlist , lengthw加曲5 )3. 读取某个vgixnip下dataset的相关信息a. 相关信息：result = eos_gd_fieldinfo(g/7w/£), fieldname, rank, dims, numbertype, dbnlist) this function retrieves information on a specific data field.b. dataset 数 h : res

32、ult = eos_gd_inqfields(g/7t/£>, fieldlist, rank, numbertype)c. 值：result = eos_gd_readfield( gridid, fieldname, buffer eqge=array ,start way , str ide way)4. 下血以打开 merra300.prod.assim.inst6_3d_ana_nv.20100531 .hdf 分析相关 datafields 和attributes 为例pro analysishdf；根据文件名和vgroup名获取信息rl 二 eos_gd_quer

33、y(' merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf , $'e0sgrid，, info)print,' info二'，info；根据文件名获取vgroup个数信息(好像有些识别不了)r2 二 eos_gd_inqgridmerra300. prod, assiin. inst6_3d_ana_nv. 20100531. hdf , gridlist , length=length)print,gridlist, length；打开一个确定的vgroup;open the filefid=e0s_g

34、d_0pen(，merra300. prod, assim. inst6_3d_ana_nv. 20100531. hdf', /read);get the gridgrid二 eos_gd_attach(fid,' eosgrii)')；读取属性数目r3 = eos_gd_inqattrs( grid, attrlist, length=lengthl)print, attri ist, lengthl;读取_fv_u属性相关信息和值rd = e0s_gd_attrinf0(grid, * _fv_u，, numbertype, count)r5 = eos_gd_r

35、eadattr(grid,'_fv_u', datbuf)print, numbertype, count, datbuf;读取datafield数目r6 = eos_gd_inqfields(grid, fieldlist, rank, numbertype) print,fieldlist, rank, numbertype;读取名为ps的datafield值和相关信息r7 = eos_gd_fieldinfo(grid,' ps', rank, dims, numbertype, dimlist) r6 = eos gd readfield(grid,&#

36、39; ps', data) print, rank, dims, numbertype, diml istend运行结果：info= missing_value time, height472ps, delp, t, u, v, qv, 03, xdim, ydimjleight, time344444141111lr"lrr0000055666600.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.000000000.00

37、0000000.000000000-11.8000000e-1. 8000000e+00890000000.54036112-1-111-120eosgrid7missingvalue1354 1. 00000c+0150. 0000000. 0000000. 000000ps, delp, t, u, v, qv, 03, xdim, ydim, height, time3444444111155555556666354036145xdim, ydim, time其中绿色字体标识的是：直接从filename和gridname读収相关信息：result = eos_

38、gd_query( filename, gridname, info)中 info 的值，参数解释如下：fieldidl data typedescriptionattributesstring arrayarray of attribute namesdimension_namesstring arraynames of dimensionsdimension_sizeslong arraysizes of dimensionsfield_namesstring arraynames of fieldsfield_rankslong arrayranks (dimensions) of fi

39、eldsfield_t ypeslong arrayidl types of fieldsgctp_projectionlonggctp projection codegctp_projection_parmdouble arraygctp projection parametersgctp_spheroidlonggctp spheroid codegctp_zonelonggctp zone code (for utm projecti on)image_lowrightdouble2location of lower right corner (meters)image_upleftdo

40、uble2location of upper left corner (meters)image_x_dimlongnumber of columns in grid imageimage_y_dimlongnumber of rows in grid imagenum_attributeslongnumber of attributesnum_dimslongnumber of dimensionsnum_idx_mapslongnumber of indexed dimension mapping entriesnum_mapslongnumber of dimension mapping

41、 entriesnum_fieldslongnumber of fieldsnum_geo_fieldslongnumber of geolocation field entriesorigin_codelongorigin codepix_reg_codelongpixel registration code附：eosgrid在hdfvicw中结构图coo>0®>0 ch 山 qos2nq_m/p_o aell® gahs2)多个vgroup下的添加hdf文件现在有问题;f ile=f ilepath (j ctscan. dat', subdir=，

42、examp les/data");openr, lun, file, /get lun;data=bytarr(256, 256);readu, lun, data;free_lun, lunfid = e0s_gd_0pen(gdq. hdf，/create)help, fidupleftpt=fltarr(2)lowrightpt=fltarr(2)upleftpt0=10584. 50041dupleftpt1=3322395. 95445dlowrightpt0=813931.10959dlowrightptl=214162. 53278dxdimsize=120ydimsi

43、ze=200grid = eos_gd_create(fid,' who', xdimsize, ydimsize, upleftpt, lowrightpt)help, gridstatus 二 eos_gd_detach(grid)gridl = eos_gd_attach(fid,'who')si = eos_gd_defdim(gridl, "ydint, 2)s2 二 eos_gd_defdim(gridl, xdin)，2)print, si, s2f32 = 3. 14s4 = eos_gd_writeattr(gridl, "

44、scalarfloat", f32)help,s4s3 = eos_gd_deffield(gridl, "temperature", "ydim, xdim，z, 5)help,s3;temperatureindgen (2,2);s2 二 eos_gd_writefield(grid, temperature, temperature)r6=eos_gd_detach (gridl)help,r6r7二eos_gd _close(fid)help,r7参考网址：/fonims/index.phphttd：/hdfco

45、sorg/fonjms/aichivc/indcxdhp?t339html问题：1.定义的vgroup显示成功了，但是在hdfview屮看不了()2. eos_gd_deffield总是定义不成功()3.3写入和读出无格式二进制文件，并利用idl的一个demo实现3维可视化1注意这里是用writeu进行写入，写入的是二进制，如果用printf写入，那就是ascii码。 pro becomedathdf id=hdf_ sd start c merra300. prod, assim. inst6 3d ana nv. 20100531. hdf')index = hdf_sd_name

46、toindex(hdfid, ' ps')if (index it 0) then $message, ' sds was not found: ' + varnameselect and read the sosvarid = hdf_sd_select(hdfid, index)hdf_sd_getdata, varid, datahdf_sd_endaccess, varidhelp,data;data=dist(10, 100);data=lindgen(10, 100)openw, lun,' today. dat/ , /get_lunwri

47、teu, lun, datafree_lun, lunend2注意writeu对应readu, prints应readf,分别是二进制和ascii的写入和读取 pro testdata_file = filepathc today. datj, $ subdirectory二'examples，, ' data');import elevation datadem data = fltarr(540, 361, 4)openr, unit, data_filc, /get_lunhelp,unitreadu, unit, dem_datafree lun, unit;i

48、n crease size of data for visibi li ty.;dem_data = congrid(dcm_data, 12& 12& /interp) ;loadct,1;tvscl, dem_data;dem data=median(dem data, 4);shade surf, dem datahelp,dem data;z=dem_data 0:255, 0:255, 1;shade_surf, zprint, dem data0:3, 0:3, 0:3;print, dem dataend注意：1. 由于writeu是以byte为单位写入，所以最多只能8个字符，也就是说如果出现 6334782.234将会出现截断：6334782.,所以最好在writeu之前进行数据的处理。2. 如果要保留精度的话也对以通过printf和readf方式来实现，在写入和读出时定义for