Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DQM database and web application #72

Merged
merged 57 commits into from
Jul 26, 2023
Merged

Conversation

jlenain
Copy link
Collaborator

@jlenain jlenain commented Jun 28, 2023

This PR introduces the web application for the Data Quality Monitoring tool, as well as all the needed recipes to setup the accompanying infrastructure (database configuration, recipes to create the two needed virtual machine on a local cloud with e.g. OpenStack).

jlenain added 6 commits June 28, 2023 10:31
After discussion with Victor at LPNHE, better not to expose the primary DB in write access to the world. Instead of pushing DQM output from DIRAC workers to the DB, better pulling the DQM FITS output files from the primary VM.
@jlenain jlenain added the enhancement New feature or request label Jun 28, 2023
@jlenain jlenain self-assigned this Jun 28, 2023
@codecov
Copy link

codecov bot commented Jun 28, 2023

Codecov Report

Patch coverage: 58.08% and project coverage change: +2.69% 🎉

Comparison is base (6af4ce8) 0.38% compared to head (b9cdf48) 3.08%.

Additional details and impacted files
@@            Coverage Diff            @@
##           master     #72      +/-   ##
=========================================
+ Coverage    0.38%   3.08%   +2.69%     
=========================================
  Files          30      36       +6     
  Lines        3404    3571     +167     
=========================================
+ Hits           13     110      +97     
- Misses       3391    3461      +70     
Files Changed Coverage Δ
src/nectarchain/dqm/bokeh_app/main.py 0.00% <0.00%> (ø)
src/nectarchain/dqm/start_calib.py 0.00% <0.00%> (ø)
src/nectarchain/dqm/db_utils.py 36.00% <36.00%> (ø)
src/nectarchain/dqm/bokeh_app/app_hooks.py 82.85% <82.85%> (ø)
src/nectarchain/dqm/__init__.py 100.00% <100.00%> (ø)
.../nectarchain/dqm/bokeh_app/tests/test_app_hooks.py 100.00% <100.00%> (ø)
src/nectarchain/dqm/tests/test_db_utils.py 100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tibaldo
Copy link
Member

tibaldo commented Jul 13, 2023

Would it be too much to ask to add a test routine for this new material before the PR is merged?

@jlenain
Copy link
Collaborator Author

jlenain commented Jul 13, 2023

Would it be too much to ask to add a test routine for this new material before the PR is merged?

Hi @tibaldo,

It would be tough to implement extensive unit tests without actually deploying the overall VM infrastructure, but I can certainly implement some, at least for the data base utilities and maybe on the bokeh side.

@jlenain
Copy link
Collaborator Author

jlenain commented Jul 21, 2023

Hi @tibaldo ,

I think I found a way to extensively test all these new features, using an in-memory data base for unit tests. This PR is ready for another review.

@jlenain jlenain requested a review from tibaldo July 21, 2023 21:47
Copy link
Member

@tibaldo tibaldo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @jlenain, thanks a lot for this impressive amount of work! Using the ZEO/ZODB servers and running tests via an in-memory instance looks like a very clever and effective solution.
I only have two minor comments on hard-coding the camera geometry to be used, but otherwise the PR looks good to be merged to me.

@jlenain jlenain merged commit a714ae5 into cta-observatory:master Jul 26, 2023
@jlenain jlenain deleted the dqm-zodb branch July 26, 2023 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants