Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for NRL v2 #42

Open
rcasey-earthscope opened this issue Feb 13, 2024 · 14 comments
Open

Support for NRL v2 #42

rcasey-earthscope opened this issue Feb 13, 2024 · 14 comments
Assignees
@is-unavco
Labels
enhancement New feature or request

Comments

@rcasey-earthscope
Copy link
Collaborator

rcasey-earthscope commented Feb 13, 2024 •
edited
Loading

Yasmine was adapted to use NRL v1, which involved a .zip file download of an indexed tree of nominal response RESP files, organized by type, manufacturer, model, and configuration settings.

Yasmine must be adapted to NRL v2, which is a suite of web services and a web tool that provide a dynamic and growing set of those same responses, now combinable on the fly, and delivered in RESP, StationXML, and StationXML-RESP formats.

NRL v2 can return a proper combination of sensor and datalogger response stacking via the web service targeting a specific instance of instrument configuration.

NRL v2 can also return original standalone responses (components) for sensor and datalogger, non combined, with the idea that these could be combined by the client from local store.

The caution for NRL v2 responses is that a sensor component will only contain a rudimentary representation of Stage 1. Datalogger components, which represent subsequent stages in a combined response, also starts its count as Stage 1, with N stages being possible from there. This is a stack shift of 1 that can be misleading, but is at the core of how the system allows combining the two components.

So, the final combination that comes from the web service, with (S)ensor and (D)atalogger components could be represented as:

Component: (S)1 + D(1) + D(2) + D(3).....D(N)
Stage Num: 1 2 3 4 0

The final stage number of a complete response in RESP terms is always 0. In StationXML, Stage 0 is referred to as the InstrumentSensitivity.

Gillian currently has a branch (nrl_v2) that can interact with NRL v2 via the Yasmine tool, but there are some integration issues that still need to be worked out. This is probably a good place to start.

A developer working on Instrumentation Services tools is working on a Python library to also permit the interfacing with NRL v2, via ObsPy, to support a number of Python tools. However, the date of completion for this is uncertain, so unsure if this will be available for use soon enough.

https://docs.fdsn.org/projects/stationxml/en/latest/ (StationXML specification)
https://ds.iris.edu/NRL/ (NRL v1) - to be retired end of April 2024
https://ds.iris.edu/ds/nrl/ (NRL v2 web page)
https://service.iris.edu/irisws/nrl/docs/1/ (NRL v2 web services)

We should make sure that the base URL for NRL v2 can be easily configured by the installer as the NRL URL will change over time.
@rcasey-earthscope rcasey-earthscope added the enhancement New feature or request label Feb 13, 2024
@metempleton
Copy link
Collaborator

I'm testing Gillian's nrl_v2 branch of yasmine and I'd like to test the "integration issues that still need to be worked out" that you mention. Can you include enough details about them for me to test them? I can't tell what they are. Thanks!

@rcasey-earthscope
Copy link
Collaborator Author

Hi Mary- I'm in the middle of some CCP code at the moment, so not prepared to drive Yasmine to get to the spot in question. However, I can best describe it by going through the steps to create a new station and channel, then use the NRL tooling to define your instrumentation.

While the component information is able to be retrieved from NRL v2, I don't think that they combine correctly. Ian has been looking into this as well, but I don't see sign of any code commits to this branch thus far.

Let us know if you see what is going on, because I remember getting some errors popping up in putting together the instrumentation. I think it has to do with the stage mismatch between sensor and datalogger, but I could be wrong.

@metempleton
Copy link
Collaborator

Thanks Rob - I've spent some time combining responses and didn't notice anything during fairly extensive response editing. The stationXML validates both with yasmine and with our validator. It's not a drop everything emergency, but I'm interested in how to reproduce what you found when there's time.

@is-unavco
Copy link
Collaborator

Thank you Rob and Mary for the additional insights. I am in the process of tracking down and patching these "integration issues", but nothing has popped up in my first pass. Mary, if you do happen to encounter any integration issues in your use of Yasmine please feel free to tag me when opening the ticket.

@metempleton
Copy link
Collaborator

Will do Ian - thanks!

Though not a ticket, one thing I've resolved is that the difference between the way v1 and v2 handle stages is addressed properly in the nrl_v2 branch. Maybe this can save you some time. I don't actually know what an "integration issue" is.

More generally, I've been filing issue tickets regarding the main branch for a few weeks now - #35-41 & #43 on. These are also relevant to the nrl_v2 branch.

@rcasey-earthscope
Copy link
Collaborator Author

rcasey-earthscope commented Mar 1, 2024 via email

@rcasey-earthscope
Copy link
Collaborator Author 

yasmine-backend   | 2024-03-01 22:02:18,791 - INFO - web - 200 GET /api/nrl/sensors/0?_dc=1709330538680 (172.18.0.2) 76.62ms
yasmine-backend   | INFO:tornado.access:200 GET /api/nrl/sensors/0?_dc=1709330538680 (172.18.0.2) 76.62ms
yasmine-backend   | 2024-03-01 22:02:18,868 - INFO - web - 200 GET /api/nrl/dataloggers/0?_dc=1709330538682 (172.18.0.2) 137.01ms
yasmine-backend   | INFO:tornado.access:200 GET /api/nrl/dataloggers/0?_dc=1709330538682 (172.18.0.2) 137.01ms
yasmine-backend   | 2024-03-01 22:03:31,417 - ERROR - web - Uncaught exception GET /api/nrl/datalogger/response/?_dc=1709330611373&keys=EQMet&keys=SMA&keys=250%20Hz (172.18.0.2)
yasmine-backend   | HTTPServerRequest(protocol='http', host='yasmine-backend:80', method='GET', uri='/api/nrl/datalogger/response/?_dc=1709330611373&keys=EQMet&keys=SMA&keys=250%20Hz', version='HTTP/1.1', remote_ip='172.18.0.2')
yasmine-backend   | Traceback (most recent call last):
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 68, in __init__
yasmine-backend   |     self.sensors = self._parse_ini(sensor_index)
yasmine-backend   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 129, in _parse_ini
yasmine-backend   |     cp = self._get_cp_from_ini(path)
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 320, in _get_cp_from_ini
yasmine-backend   |     with codecs.open(path, mode='r', encoding='UTF-8') as f:
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "", line 918, in open
yasmine-backend   | FileNotFoundError: [Errno 2] No such file or directory: '/opt/yasmine/data/_media/nrl/content/NRL/sensors/index.txt'
yasmine-backend   |
yasmine-backend   | During handling of the above exception, another exception occurred:
yasmine-backend   |
yasmine-backend   | Traceback (most recent call last):
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
yasmine-backend   |     result = await result
yasmine-backend   |              ^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/tornado/gen.py", line 780, in run
yasmine-backend   |     yielded = self.gen.throw(exc)
yasmine-backend   |               ^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/handlers/base.py", line 79, in get
yasmine-backend   |     res = yield self.async_call(self.async_get, *args, **kwargs)
yasmine-backend   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/tornado/gen.py", line 767, in run
yasmine-backend   |     value = future.result()
yasmine-backend   |             ^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
yasmine-backend   |     result = self.fn(*self.args, **self.kwargs)
yasmine-backend   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/handlers/base.py", line 63, in async_call
yasmine-backend   |     return func(*args, **kwargs)
yasmine-backend   |            ^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/handlers/xml_nrl.py", line 57, in async_get
yasmine-backend   |     return self.helper.get_datalogger_response_str(self.get_arguments('keys'))
yasmine-backend   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/helpers/nrl/nrl_helper.py", line 71, in get_datalogger_response_str
yasmine-backend   |     path = self._build_path(self.nrl.dataloggers, datalogger_keys)
yasmine-backend   |                             ^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/helpers/nrl/nrl_helper.py", line 118, in nrl
yasmine-backend   |     self._nrl = NRL(os.path.join(self.content_folder,'NRL'))
yasmine-backend   |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 312, in __init__
yasmine-backend   |     super(self.__class__, self).__init__()
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 76, in __init__
yasmine-backend   |     self.sensors = self._parse_ini(sensor_index)
yasmine-backend   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 129, in _parse_ini
yasmine-backend   |     cp = self._get_cp_from_ini(path)
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 320, in _get_cp_from_ini
yasmine-backend   |     with codecs.open(path, mode='r', encoding='UTF-8') as f:
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "", line 918, in open
yasmine-backend   | FileNotFoundError: [Errno 2] No such file or directory: '/opt/yasmine/data/_media/nrl/content/NRL/sensor/index.txt'
yasmine-backend   | ERROR:tornado.application:Uncaught exception GET /api/nrl/datalogger/response/?_dc=1709330611373&keys=EQMet&keys=SMA&keys=250%20Hz (172.18.0.2)
yasmine-backend   | HTTPServerRequest(protocol='http', host='yasmine-backend:80', method='GET', uri='/api/nrl/datalogger/response/?_dc=1709330611373&keys=EQMet&keys=SMA&keys=250%20Hz', version='HTTP/1.1', remote_ip='172.18.0.2')
yasmine-backend   | Traceback (most recent call last):
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 68, in __init__
yasmine-backend   |     self.sensors = self._parse_ini(sensor_index)
yasmine-backend   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 129, in _parse_ini
yasmine-backend   |     cp = self._get_cp_from_ini(path)
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 320, in _get_cp_from_ini
yasmine-backend   |     with codecs.open(path, mode='r', encoding='UTF-8') as f:
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "", line 918, in open
yasmine-backend   | FileNotFoundError: [Errno 2] No such file or directory: '/opt/yasmine/data/_media/nrl/content/NRL/sensors/index.txt'
yasmine-backend   |
yasmine-backend   | During handling of the above exception, another exception occurred:
yasmine-backend   |
yasmine-backend   | Traceback (most recent call last):
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/tornado/web.py", line 1790, in _execute
yasmine-backend   |     result = await result
yasmine-backend   |              ^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/tornado/gen.py", line 780, in run
yasmine-backend   |     yielded = self.gen.throw(exc)
yasmine-backend   |               ^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/handlers/base.py", line 79, in get
yasmine-backend   |     res = yield self.async_call(self.async_get, *args, **kwargs)
yasmine-backend   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/tornado/gen.py", line 767, in run
yasmine-backend   |     value = future.result()
yasmine-backend   |             ^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
yasmine-backend   |     result = self.fn(*self.args, **self.kwargs)
yasmine-backend   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/handlers/base.py", line 63, in async_call
yasmine-backend   |     return func(*args, **kwargs)
yasmine-backend   |            ^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/handlers/xml_nrl.py", line 57, in async_get
yasmine-backend   |     return self.helper.get_datalogger_response_str(self.get_arguments('keys'))
yasmine-backend   |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/helpers/nrl/nrl_helper.py", line 71, in get_datalogger_response_str
yasmine-backend   |     path = self._build_path(self.nrl.dataloggers, datalogger_keys)
yasmine-backend   |                             ^^^^^^^^
yasmine-backend   |   File "/opt/yasmine/src/yasmine/app/helpers/nrl/nrl_helper.py", line 118, in nrl
yasmine-backend   |     self._nrl = NRL(os.path.join(self.content_folder,'NRL'))
yasmine-backend   |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 312, in __init__
yasmine-backend   |     super(self.__class__, self).__init__()
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 76, in __init__
yasmine-backend   |     self.sensors = self._parse_ini(sensor_index)
yasmine-backend   |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 129, in _parse_ini
yasmine-backend   |     cp = self._get_cp_from_ini(path)
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "/usr/local/lib/python3.12/site-packages/obspy/clients/nrl/client.py", line 320, in _get_cp_from_ini
yasmine-backend   |     with codecs.open(path, mode='r', encoding='UTF-8') as f:
yasmine-backend   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
yasmine-backend   |   File "", line 918, in open
yasmine-backend   | FileNotFoundError: [Errno 2] No such file or directory: '/opt/yasmine/data/_media/nrl/content/NRL/sensor/index.txt'
yasmine-backend   | 2024-03-01 22:03:31,446 - ERROR - web - 500 GET /api/nrl/datalogger/response/?_dc=1709330611373&keys=EQMet&keys=SMA&keys=250%20Hz (172.18.0.2) 38.35ms
yasmine-backend   | ERROR:tornado.access:500 GET /api/nrl/datalogger/response/?_dc=1709330611373&keys=EQMet&keys=SMA&keys=250%20Hz (172.18.0.2) 38.35ms

@rcasey-earthscope
Copy link
Collaborator Author

Of note is that the sensors.json and dataloggers.json are found in /Users/rob/Desktop/yasmine/install_run_Jan_2024/yasmine-stationxml-editor-nrl_v2/data/_media/nrl. I am able to make selections from a list, so the navigation is not a problem. However, the call to /api/nrl/datalogger/response seems to not yield anything.

When looking at the nrlv2 diff, I note that it looks like the entire .zip file of NRLv2 is downloaded from the web service. I guess this is fine for a first go, but I thought that Yasmine was going to call to the web service itself. Is there some kind of initialization I need to do?

@metempleton
Copy link
Collaborator

Thanks Rob!

My installation must be different, as I succeeded getting the EQMet SMA 250 Hz response with this console message:

yasmine-backend | 2024-03-01 22:47:55,154 - INFO - web - 200 GET /api/nrl/datalogger/response/?_dc=1709333274755&keys=EQMet&keys=SMA&keys=250%20Hz (172.18.0.2) 388.14ms
yasmine-backend | INFO:tornado.access:200 GET /api/nrl/datalogger/response/?_dc=1709333274755&keys=EQMet&keys=SMA&keys=250%20Hz (172.18.0.2) 388.14ms

FWIW, Gillian's branch pulls the full cached .zip file from the NRL ws and this takes time (X min?) to download and unzip. But it only does this once per day; I'm not sure what time. If yasmine is pulling a fresh copy, you'll need to wait until the following "docker-compose up" console message reads:

yasmine-backend | INFO:yasmine.app.helpers.nrl.nrl_helper:Creating an NRL key files

before proceeding, and there is nothing to tell you that this needs to be done currently.

You can also check whether the NRL has completed downloading by listing yasmine-stationxml-editor/data/_media/nrl/content/NRL - the datalogger and sensor dirs.

@gillian-earthscope
Copy link

gillian-earthscope commented Mar 2, 2024 via email

@gillian-earthscope
Copy link

gillian-earthscope commented Mar 2, 2024 via email

@rcasey-earthscope
Copy link
Collaborator Author

rcasey-earthscope commented Mar 4, 2024 via email

@rcasey-earthscope
Copy link
Collaborator Author

rcasey-earthscope commented Mar 5, 2024 via email

@rcasey-earthscope
Copy link
Collaborator Author

Since the NRL v2 download of the zip file is working, we want to review that NRL navigation is functioning correctly and that responses are being generated properly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@is-unavco is-unavco

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@metempleton @rcasey-earthscope @gillian-earthscope @is-unavco