WOUDC Data Registry is a platform that manages ozone and ultraviolet radiation data in support of the World Ozone and Ultraviolet Radiation Data Centre (WOUDC), one of six World Data Centres as part of the Global Atmosphere Watch programme of the WMO.
- Python 3 and above
- virtualenv
- Elasticsearch (5.5.0 and above)
- woudc-extcsv
Dependencies are listed in requirements.txt. Dependencies are automatically installed during installation.
# setup virtualenv
python3 -m venv woudc-data-registry_env
cd woudc-data-registry_env
source bin/activate
# clone woudc-extcsv and install
git clone https://github.com/woudc/woudc-extcsv.git
cd woudc-extcsv
pip install -r requirements.txt
python3 setup.py install
cd ..
# clone codebase and install
git clone https://github.com/woudc/woudc-data-registry.git
cd woudc-data-registry
python3 setup.py build
python3 setup.py install
# for PostgreSQL backends
pip install -r requirements-pg.txt
# set system environment variables
cp default.env foo.env
vi foo.env # edit database connection parameters, etc.
. foo.env# create database
make ENV=foo.env createdb
# drop database
make ENV=foo.env dropdb
# show configuration
woudc-data-registry admin config
# initialize model (database tables)
woudc-data-registry admin registry setup
# initialize search engine
woudc-data-registry admin search setup
# load core metadata
woudc-data-registry admin init -d data/
# cleanups
# re-initialize model (database tables)
woudc-data-registry admin registry teardown
woudc-data-registry admin registry setup
# re-initialize search engine
woudc-data-registry admin search teardown
woudc-data-registry admin search setupTIP: autocompletion can be made available in some shells via:
eval "$(_WOUDC_DATA_REGISTRY_COMPLETE=source woudc-data-registry)"# list all instances of foo (where foo is one of:
# project|dataset|contributor|country|station|instrument|deployment)
woudc-data-registry <foo> list
# e.g.
woudc-data-registry contributor list
# show a specific instance of foo with a given registry identifier
woudc-data-registry <foo> show <identifier>
# e.g.
woudc-data-registry station show 023
woudc-data-registry instrument show ECC:2Z:4052:002:OzoneSonde
# add a new instance of foo (contributor|country|station|instrument|deployment)
woudc-data-registry <foo> add <options>
# e.g.
woudc-data-registry deployment add -s 001 -c MSC:WOUDC
woudc-data-registry contributor add -id foo -n "Contributor name" -c Canada -w IV -u https://example.org -e [email protected] -f foouser -g -75,45
# update an existing instance of foo with a given registry identifier
woudc-data-registry <foo> update -id <identifier> <options>
# e.g.
woudc-data-registry station update -n "New station name"
woudc-data-registry deployment update --end-date 'Deployment end date'
# delete an instance of foo with a given registry identifier
woudc-data-registry <foo> delete <identifier>
# e.g.
woudc-data-registry deployment delete 018:MSC:WOUDC
# for more information about options on operation (add|update):
woudc-data-registry <foo> <operation> --help
# e.g.
woudc-data-registry instrument update --help# ingest directory of files (walks directory recursively)
woudc-data-registry data ingest /path/to/dir
# ingest single file
woudc-data-registry data ingest foo.dat
# ingest without asking permission checks
woudc-data-registry data ingest foo.dat -y
# verify directory of files (walks directory recursively)
woudc-data-registry data verify /path/to/dir
# verify single file
woudc-data-registry data verify foo.dat
# verify core metadata only
woudc-data-registry data verify foo.dat -l
# ingest with only core metadata checks
woudc-data-registry data ingest /path/to/dir -l# sync data registry to elasticsearch
# sync the data and metadata tables
woudc-data-registry admin search sync
# sync the data product tables
woudc-data-registry admin search product-sync# Teardown and generate entire uv_index_hourly table
woudc-data-registry product uv-index generate /path/to/archive/root
# Only generate uv_index_hourly records within year range
woudc-data-registry product uv-index update -sy start-year -ey end-year /path/to/archive/root# Teardown and generate entire totalozone table
woudc-data-registry product totalozone generate /path/to/archive/root# Teardown and generate entire ozonesonde table
woudc-data-registry product ozonesonde generate /path/to/archive/rootThe woudc-data-registry data ingest command accepts a -r/--report flag, which is a path pointing to a directory.
When that flag is provided, an operator report and a run report are automatically written to that directory
while the files are being processing.
woudc-data-registry data ingest /path/to/dir -r /path/to/reports/location
The run report has a filename run_report. The file contains a series of blocks,
one per contributor in a processing run, of the following format:
<contributor acronym>
<status>: <filepath>
<status>: <filepath>
<status>: <filepath>
...
Where <status> is either Pass or Fail, depending on how the file reported in that line fared in processing.
The operator report is a more in-depth error log in CSV format, with a filename like operator-report-<date>.csv.
Operator reports contain one line per error or warning that happened during the processing run. The operator report
is meant to be a human-readable log which makes specific errors easy to find and diagnose.
To generate emails for contributors:
woudc-data-registry data generate-emails /path/to/dir# install dev requirements
pip install -r requirements-dev.txt# build local copy of https://woudc.github.io/woudc-data-registry
cd docs
make html
python3 -m http.server # view on http://localhost:8000/# run tests like this:
cd woudc_data_registry/tests
python3 test_data_registry.py
# or this:
python3 setup.py test
# measure code coverage
coverage run --source=woudc_data_registry -m unittest woudc_data_registry.tests.test_data_registry
coverage report -mAll bugs, enhancements and issues are managed on GitHub.