Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix live workflow and add tests #177

Merged
merged 7 commits into from
Oct 8, 2024
Merged

Fix live workflow and add tests #177

merged 7 commits into from
Oct 8, 2024

Conversation

SimonHeybrock
Copy link
Member

Requires scipp/essreduce#115 for CI to pass.

@YooSunYoung
Copy link
Member

There is a typo here:

esssans/pyproject.toml

Lines 59 to 60 in b0bd383

[project.entry-points."beamlime.workflow_plugin"]
ess-loki-monitor = "ess.loki.live:LokiMonitorWorkflow"

The workflow class name should be LoKi not Loki.

@SimonHeybrock
Copy link
Member Author

SimonHeybrock commented Oct 4, 2024

There is a typo here:

esssans/pyproject.toml

Lines 59 to 60 in b0bd383

[project.entry-points."beamlime.workflow_plugin"]
ess-loki-monitor = "ess.loki.live:LokiMonitorWorkflow"

The workflow class name should be LoKi not Loki.

I don't think we are camelcasing Loki anywhere in this package? ... oh, but we do, will fix the classname.

@YooSunYoung
Copy link
Member

YooSunYoung commented Oct 4, 2024

The test is failing just because the new essreduce implementation is not released right?
When I tested it with scipp/beamlime#242, it was complaining like this:

AttributeError: 'JSONGroup' object has no attribute 'get'
Traceback (most recent call last):
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/bin/beamlime", line 8, in <module>
    sys.exit(main())
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/__main__.py", line 18, in main
    run_standalone_prototype(factory, arg_name_space=arg_parser.parse_args())
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/executables/prototypes.py", line 113, in run_standalone_prototype
    app.run()
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/applications/base.py", line 277, in run
    loop.run_until_complete(asyncio.gather(*self.tasks))
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/applications/base.py", line 241, in run_daemon
    async for message in daemon.run():
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/applications/base.py", line 150, in run
    await self.route(self.message_pipe.get())
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/applications/base.py", line 124, in route
    results.extend(self._collect_results(handler(message)))
  File "/Users/sunyoungyoo/ESS/beamlime/src/beamlime/applications/handlers.py", line 175, in reduce_data
    results = self.workflow(nxevent_data=nxevent_data, nxlog=nxlog)
  File "/Users/sunyoungyoo/ESS/esssans/src/ess/loki/live.py", line 120, in __call__
    results = self._streamed.add_chunk({JSONEventData: nxevent_data})
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/ess/reduce/streaming.py", line 208, in add_chunk
    return self._finalize_workflow.compute(self._target_keys)
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/sciline/pipeline.py", line 90, in compute
    return self.get(tp, **kwargs).compute()
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/sciline/task_graph.py", line 110, in compute
    results = self._scheduler.get(self._graph, list(targets))
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/sciline/scheduler.py", line 105, in get
    return self._dask_get(dsk, list(map(_to_dask_key, keys)))
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/dask/threaded.py", line 91, in get
    results = get_async(
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/dask/local.py", line 513, in get_async
    raise_exception(exc, tb)
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/dask/local.py", line 321, in reraise
    raise exc
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/dask/local.py", line 226, in execute_task
    result = _execute_task(task, data)
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/dask/core.py", line 133, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/dask/utils.py", line 80, in apply
    return func(*args)
  File "/Users/sunyoungyoo/ESS/esssans/src/ess/loki/live.py", line 56, in load_json_transmission_monitor_data
    return nexus_types.NeXusMonitorData[SampleRun, Incident](group[()])
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/scippnexus/base.py", line 399, in __getitem__
    dg = self._nexus.read_children(sel)
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/scippnexus/base.py", line 298, in _nexus
    self._populate_fields()
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/scippnexus/base.py", line 313, in _populate_fields
    self._lazy_nexus = self._definitions.get(self.attrs.get('NX_class'), NXobject)(
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/scippnexus/_cache.py", line 51, in __get__
    val = self.func(instance)
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/scippnexus/base.py", line 233, in attrs
    return MappingProxyType(Attrs(self._group.attrs))
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/scippnexus/attrs.py", line 13, in __init__
    self._attrs = dict(attrs)
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/_collections_abc.py", line 886, in __iter__
    yield from self._mapping
  File "/Users/sunyoungyoo/miniconda3/envs/lime-dev-310/lib/python3.10/site-packages/ess/reduce/nexus/json_nexus.py", line 221, in __iter__
    if (attrs := self._node.get('attributes')) is not None:

@SimonHeybrock
Copy link
Member Author

SimonHeybrock commented Oct 7, 2024

Sounds like there was as nested creation of JSONGroup ("constructing" it twice rather than once) in Beamlime? I will look into it (edit: yes, fixed in scipp/beamlime#242).

@SimonHeybrock SimonHeybrock merged commit f5bc5c5 into main Oct 8, 2024
4 checks passed
@SimonHeybrock SimonHeybrock deleted the fix-live branch October 8, 2024 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants