Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement openmetrics endpoint in mula #662

Merged
merged 49 commits into from
May 5, 2023
Merged

Implement openmetrics endpoint in mula #662

merged 49 commits into from
May 5, 2023

Conversation

jpbruinsslot
Copy link
Contributor

@jpbruinsslot jpbruinsslot commented Apr 5, 2023 •
edited
Loading

Example output:

$ curl http://localhost:8004/metrics

# HELP app_settings_info Scheduler configuration settings
# TYPE app_settings_info gauge
app_settings_info{katalogus_cache_ttl="30",monitor_organisations_interval="60",pq_maxsize="1000",pq_populate_grace_period="86400",pq_populate_interval="60",pq_populate_max_random_objects="50"} 1.0
# HELP scheduler_qsize Size of the scheduler queue
# TYPE scheduler_qsize gauge
scheduler_qsize{scheduler_id="normalizer-dev"} 0.0
scheduler_qsize{scheduler_id="boefje-dev"} 0.0

@jpbruinsslot
Implement a metrics endpoint
f668b4d
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
ae0e08d
@jpbruinsslot
Remove comments
328b096
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
c45f817 
* main:
  Add sudo in Debian install manual (#153)
  Fix report translation (#609)
  Feature/add organization dashboard (#481)
  [CRT_SH] 🐛 don't treat invalid certificates from 3rd party as active (#541)
  Use PostgreSQL 15 in a single container (#546)
  [CRT_SH] 🐛 don't treat invalid certificates from 3rd party as active (#610)
  Update mula design documentation (#478)
  pass on underscore domains (#550)
  Feature: Add Subfinder Boefje (#516)
  add project discovery Nuclei boefje (#518)
  Remove loop from get random objects (#558)
  Bump urllib3 from 1.26.14 to 1.26.15 (#576)
  Bump black from 22.3.0 to 23.1.0 (#578)
  Bump iniconfig from 1.1.1 to 2.0.0 (#579)
  Add indices for Bytes (#600)
  refactor(organization lists): unify access to organizations by user (#528)
@jpbruinsslot
Start with openmetrics implementation
b13f822
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
a5b5c69 
* main:
  Add 1.6 and 1.7 release notes (#630)
  Change mula Debian package to always use PostgreSQL (#597)
  Fixes the redirect to the onboarding flow (#621)
  changes to allow for custom fonts (#593)
  Bump pre-commit from 3.2.0 to 3.2.1 (#575)
  Feature/mula new boefjes (#574)
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
96b14c7 
* main:
  Replaced `psycopg2-binary` with `psycopg2` (#638)
@jpbruinsslot
Restructure metrics usage
63d563e
@jpbruinsslot
Restructure metrics
0eb2094
@jpbruinsslot
Format with black
9b3d030
@jpbruinsslot jpbruinsslot added the mula Issues related to the scheduler label Apr 5, 2023
@jpbruinsslot jpbruinsslot self-assigned this Apr 5, 2023
@jpbruinsslot jpbruinsslot marked this pull request as ready for review April 5, 2023 10:19
@jpbruinsslot jpbruinsslot requested a review from a team as a code owner April 5, 2023 10:19
praseodym
praseodym suggested changes Apr 5, 2023
View reviewed changes
mula/scheduler/app.py Outdated Show resolved Hide resolved
mula/scheduler/context/context.py Outdated Show resolved Hide resolved
dekkers
dekkers requested changes Apr 13, 2023
View reviewed changes
mula/scheduler/context/context.py Outdated Show resolved Hide resolved
mula/scheduler/context/context.py Outdated Show resolved Hide resolved
mula/setup.py Outdated Show resolved Hide resolved
praseodym and others added 12 commits April 13, 2023 17:51
@praseodym
Merge branch 'main' into feature/mula-metrics-endpoint
ae9319e
@jpbruinsslot
Remove unused Info
dd0eb8b
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
5f6a28e 
* main:
  Update `pre-commit` dependencies and enable Ruff autofix (#739)
  Cleanup Rocky requirements (#729)
  Add (I) to ruff and fix imports (#723)
  Add port-common bit and KAT-OPEN-COMMON-PORT FindingType (#734)
  Use setuptools-scm to write correct version to version.py (#737)
  Enable and disable bits (#732)
  Scan level inheritance chain (#722)
  Octopoes origin param endpoint (#731)
  Do not propagate scan level from DNSSPFMechanismHostname (#721)
  Fix: Unnecessary white space within links (#728)
  chore/refactor katalogus settings setup (#542)
@jpbruinsslot
Format with black
cf6cd2a
@jpbruinsslot
Fix formatting
04dcdfd
@jpbruinsslot
Fix formatting
c0d0502
@jpbruinsslot
Fix formatting
5c44405
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
9366f02
@jpbruinsslot
Remove katalogus metrics
9e4c273
@jpbruinsslot
Add katalogus cache ttl in application info metric
0d7eb30
@dekkers
Revert formatting changes
614b5a7
@jpbruinsslot
Merge branch 'main' into feature/mula-metrics-endpoint
ee1a94c
Darwinkel
Darwinkel previously approved these changes May 2, 2023
View reviewed changes
Copy link
Contributor

@Darwinkel Darwinkel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checklist for QA:

  • I have checked out this branch, and successfully ran a fresh make kat.
  • I confirmed that there are no unintended functional regressions in this branch:
    • I have managed to pass the onboarding flow
    • Objects and Findings are created properly
    • Tasks are created and completed properly
  • I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

  • All organisations are represented with a normalizer and boefje entry
  • Queue sizes are correctly updated for both boefje and normalizer
  • App settings are correctly updated after .env changes

@Darwinkel
Copy link
Contributor

Ready for merge after fixing the merge conflict.

@jpbruinsslot jpbruinsslot mentioned this pull request May 2, 2023
Open
@dekkers dekkers merged commit 9175043 into main May 5, 2023
@dekkers dekkers deleted the feature/mula-metrics-endpoint branch May 5, 2023 12:39
jpbruinsslot added a commit that referenced this pull request May 8, 2023
@jpbruinsslot
Merge branch 'main' into feature/optimize-scheduler
89da02b 
* main: (39 commits)
  Add delete oois permission (#783)
  Implement openmetrics endpoint in mula (#662)
  Rabbitmq health checks for Bytes integration tests instead of sleep() (#920)
  Groups and Permissions for Organization member (#606)
  Wrap all import errors in ModuleExceptions while resolving plugins (#818)
  Feature/octopoes path queries (#851)
  Fix reset scan profile bug and reduce octopoes calls in view (#910)
  Fix fierce normalizer to create hostnames without a trailing dot (#909)
  Updated playwright (#903)
  Use superuser credentials from .env file (#860)
  Localize account scoped url's (#861)
  Dependabot merges (#902)
  Updated linter configs (#834)
  Fix cancel url at organization edit view (#850)
  Add OpenTelemetry tracing to Mula, Bytes, Keiko, and KATalogus (#831)
  Normalizers should be defined as generators (#830)
  Rdo workflows (#816)
  Updated linter configs (#825)
  Add `rocky_version` to footer (#821)
  Update PR template with PGP-sign checkbox and absolute URL's (#826)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Donnype Donnype Donnype left review comments

@underdarknl underdarknl underdarknl left review comments

@dekkers dekkers dekkers approved these changes

@Darwinkel Darwinkel Darwinkel left review comments

@praseodym praseodym praseodym left review comments

Assignees

@jpbruinsslot jpbruinsslot

Labels
mula Issues related to the scheduler
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Mula metrics endpoint for current katalogus info
7 participants
@jpbruinsslot @Donnype @Darwinkel @praseodym @dekkers @underdarknl @TwistMeister