Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve metadata in functions and component file #139

Closed
10 tasks done
veenstrajelmer opened this issue Aug 11, 2023 · 0 comments · Fixed by #218
Closed
10 tasks done

Improve metadata in functions and component file #139

veenstrajelmer opened this issue Aug 11, 2023 · 0 comments · Fixed by #218

Comments

@veenstrajelmer
Copy link
Collaborator

veenstrajelmer commented Aug 11, 2023

TODO:

  • two header examples in Pass metadata between objects #93
  • first improvements in Get nodalfactors/xfac settings from component file #144
  • how to make computer-readable (and writeable) analysis settings for component file (necessary since we will run with fu_alltimes soon)
  • remove commented line in metadata_from_obj >> no, this would make using noos files or netcdf files way more complicated.
  • remove metadata_from_ddlmeta(), not used/tested (also not in kwk repos) and seems not generic
  • avoid writing diafile with tzone!=1 (already done). Mention somewhere "in DONAR zijn alle data in MET" (so prevent writing with other tzone should not be possible)
  • avoid writing compfile with nodalfactors=False or fu_alltimes=True or source!="schureman" until the metadata can be included in header. Even then, we still cannot remove stats_xfac0 from example scripts since we need it when starting from diafiles
  • make source (schureman/foreman) also attribute to be added and checked everywhere
  • update metadata for extremes in calc_HWLW? At least TYP >> not necessary since we distinguish with HWLWcode columns in dataframe. Could be done with additional groepering attribute, but would make it too complex.
  • prevent prediction with xfac=True with compfile with xfac=False (or vice versa). Compare all ana/pred options in prediction() function.

Non-parsed metadata from component file (example headers in #93):

  • *valid for year
  • *SA/SM from different period
  • *theoretische f-factoren (means xfac=0)
  • *component splitting
  • CODE=3 >> not relevant for hatyan2

Missing metadata in component file:

  • timestep_min (PERD start stop 60 is timezone according to anadea.f)
  • nodalfactors on/off (always on)
  • xfac (off if "theoretische" in headerlines)
  • nodalfactors on all timesteps or per year (always per year up to now)
  • MIDD replaced by slotgemiddelde

Missing metadata in diafile:

  • timezone, but donar is always MET

Current status is:

* written with hatyan-2.7.2
* origin :  from timeseries dia file
* nodalfactors :  True
* xfac :  True
* fu_alltimes :  False
* groepering :  NVT
* timestep_min :  60.0
* timestep_unit :  min
* TYP :  TE
* components_sec :  SA, SM imported from component file
STAT  VLISSGN    WATHTE    NAP    cm    1
PERD  20090001  000000  20120031  230000     60
COMP      3
MIDD      1.00
NCOM    95
COMP    0     0.000000     1.000    0.00  A0          
[...]

MWE:

import os
import hatyan

dir_testdata = 'C:\\DATA\\hatyan_data_acceptancetests'
current_station = 'VLISSGN'

#read timeseries, components from analysis, write component file
file_data_comp0 = [os.path.join(dir_testdata, 'predictie2019', f'{current_station}_obs{i}.txt') for i in [1,2,3,4]]
ts_measurements_group0 = hatyan.readts_dia(filename=file_data_comp0, station=current_station)
comp_frommeas_avg_group0 = hatyan.analysis(ts=ts_measurements_group0, const_list='year')
hatyan.write_components(comp_frommeas_avg_group0, 'test_frommeas.cmp')

#read+write component file
file_data_comp1 = os.path.join(dir_testdata, 'predictie2019', f'{current_station}_ana.txt')
COMP_merged = hatyan.read_components(filename=file_data_comp1)
COMP_merged.loc['A0','A'] = 1.23 #TODO: add to metadata if overwritten
hatyan.write_components(COMP_merged, 'test_fromfile.cmp')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant