Feature/bytes openmetrics

[Donnype]

Changes

Add a /metrics endpoint that can be consumed by Prometheus.

Up for discussion at this point:

• For #rawfiles per organization I started out with a file scan but added a database scan after some discussion. 2 million files takes roughly half a second for the file system scan and 1.3 seconds from the database (likely due to the join). We should decide whether to keep one of them or both (also see the discussion in the ticket).
• I did several experiments with caching the methods, but the cleanest solution if we want to add caching would either be using the cachetools library or suggest caching at the proxy layer for the metric endpoint. I eventually left out caching to discuss it further here.

Issue link

Fixes #298

Proof

Metrics example output generated in the docs (any inconsistencies were probably created during manual testing and generating files):

# HELP bytes_database_organizations_total Total amount of organizations in the bytes database.
# TYPE bytes_database_organizations_total gauge
bytes_database_organizations_total 11.0
# HELP bytes_database_raw_files_total Total amount of raw files in the bytes database.
# TYPE bytes_database_raw_files_total gauge
bytes_database_raw_files_total{organization_id="a"} 286296.0
bytes_database_raw_files_total{organization_id="ab"} 159.0
bytes_database_raw_files_total{organization_id="abc"} 183.0
bytes_database_raw_files_total{organization_id="b"} 286075.0
bytes_database_raw_files_total{organization_id="ba"} 101.0
bytes_database_raw_files_total{organization_id="c"} 286074.0
bytes_database_raw_files_total{organization_id="d"} 286074.0
bytes_database_raw_files_total{organization_id="def"} 9.0
bytes_database_raw_files_total{organization_id="e"} 286073.0
bytes_database_raw_files_total{organization_id="f"} 285071.0
bytes_database_raw_files_total{organization_id="g"} 285069.0

---

Checklists for authors:

Code Checklist

• This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
• I have written unit tests for the changes or fixes I made.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have made corresponding changes to the documentation, if necessary.

---

Checklist for code reviewers:

• The code does not violate Model-View-Template and our other architectural principles.
• The code prioritizes readability over performance where appropriate.
• The code does not bypass authentication or security mechanisms.
• The code does not introduce any dependency on a library that has not been properly vetted.
• The code contains docstrings, comments, and documentation where needed.

---

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make kat.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

• bullet point + screenshot (if useful) per tested functionality

What doesn't work:

• bullet point + screenshot (if useful) per tested functionality

Bug or feature?:

• bullet point + screenshot (if useful) if it is unclear whether something is a bug or an intended feature.

[github-actions]

File	Coverage
All files	65%	❌
bits/definitions.py	64%	❌
bits/runner.py	56%	❌
bits/https_availability/https_availability.py	93%	✅
bits/oois_in_headers/oois_in_headers.py	57%	❌
bits/spf_discovery/internetnl_spf_parser.py	55%	❌
bits/spf_discovery/spf_discovery.py	72%	❌
octopoes/api/api.py	59%	❌
octopoes/api/models.py	75%	✅
octopoes/api/router.py	52%	❌
octopoes/core/app.py	69%	❌
octopoes/core/service.py	46%	❌
octopoes/events/events.py	96%	✅
octopoes/events/manager.py	65%	❌
octopoes/models/__init__.py	86%	✅
octopoes/models/datetime.py	66%	❌
octopoes/models/exception.py	83%	✅
octopoes/models/origin.py	70%	❌
octopoes/models/path.py	99%	✅
octopoes/models/types.py	95%	✅
octopoes/models/ooi/certificate.py	96%	✅
octopoes/models/ooi/email_security.py	95%	✅
octopoes/models/ooi/findings.py	94%	✅
octopoes/models/ooi/network.py	97%	✅
octopoes/models/ooi/service.py	91%	✅
octopoes/models/ooi/software.py	71%	❌
octopoes/models/ooi/web.py	81%	✅
octopoes/models/ooi/dns/records.py	95%	✅
octopoes/models/ooi/dns/zone.py	77%	✅
octopoes/repositories/ooi_repository.py	40%	❌
octopoes/repositories/origin_parameter_repository.py	53%	❌
octopoes/repositories/origin_repository.py	53%	❌
octopoes/repositories/scan_profile_repository.py	45%	❌
octopoes/xtdb/client.py	39%	❌
octopoes/xtdb/query_builder.py	69%	❌
octopoes/xtdb/related_field_generator.py	73%	❌
tests/conftest.py	91%	✅

Minimum allowed coverage is 75%

Generated by 🐒 cobertura-action against b30eac7

[praseodym]

Should we set BYTES_METRICS_TTL_SECONDS to a reasonable default, such as 300s?

[Donnype]

@praseodym Perhaps, but perhaps our common case now is development where you would prefer to get more direct feedback pinging the metrics endpoint?

[Donnype]

I added some extra documentation about performance tuning.

[Darwinkel]

There is currently only scarce documentation on the authentication mechanism for the Bytes API, meaning that metrics consumers are kind of left to figure it out for themselves. There are no instructions or examples as to how one can actually use /metrics. As many of our other module API's don't bother with authentication, we could consider removing it.

[dekkers]

@praseodym Perhaps, but perhaps our common case now is development where you would prefer to get more direct feedback pinging the metrics endpoint?

If we need a different setting in development we should put that in the .env file. In my opinion all our defaults in the code should be set to sensible defaults so that we minimize the number of env variables are user needs to change.

[Donnype]

@dekkers With the current docs suggesting running make env as well the env-dist defaults also end up in some production environments. Determining the sensible default for both development and production (given we can comfortably handle a 30s interval ping for 2 million rows) is perhaps a bit subjective, but since there might not be a lot of development on the metrics endpoint I'm fine with setting it to 300 seconds.

[Donnype]

@Darwinkel #805 (comment)