perf(crisis_room): optimize loading of aggregates

[Lisser]

Changes

This PR optimizes loading of aggregates on the cross-organization crisis-room.

Note: This PR only addresses the cross-organization crisis-room. Follow-up work is needed to also improve the per-organization crisis-room

By:

• adding an endpoint in Octopoes {org}/finding_types/count
• adding a corresponding method in OctopoesAPIConnector
• refactoring CrisisRoomView

Removed in CrisisRoomView refactor:

• old flow that pulled all findings in memory
• stop using the suboptimal build_findings_list_from_store flow that queried the OOI Information table for each FindingType

Replaced by:

• fetching the counts per finding type, per organization
• fetching the OOI Information records for the unique FindingTypes
• merging the data in a short-lived dataclass OrganizationFindingTypeCount to render in the template

Issue link

#797

Proof

---

Checklists for authors:

Code Checklist

• This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
• I have written unit tests for the changes or fixes I made.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have made corresponding changes to the documentation, if necessary.

---

Checklist for code reviewers:

• The code does not violate Model-View-Template and our other architectural principles.
• The code prioritizes readability over performance where appropriate.
• The code does not bypass authentication or security mechanisms.
• The code does not introduce any dependency on a library that has not been properly vetted.
• The code contains docstrings, comments, and documentation where needed.

---

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make kat.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

• bullet point + screenshot (if useful) per tested functionality

What doesn't work:

• bullet point + screenshot (if useful) per tested functionality

Bug or feature?:

• bullet point + screenshot (if useful) if it is unclear whether something is a bug or an intended feature.

[github-actions]

File	Coverage
All files	65%	❌
bits/definitions.py	64%	❌
bits/runner.py	56%	❌
bits/https_availability/https_availability.py	93%	✅
bits/oois_in_headers/oois_in_headers.py	57%	❌
bits/spf_discovery/internetnl_spf_parser.py	55%	❌
bits/spf_discovery/spf_discovery.py	72%	❌
octopoes/api/api.py	89%	✅
octopoes/api/models.py	75%	✅
octopoes/api/router.py	52%	❌
octopoes/core/app.py	69%	❌
octopoes/core/service.py	46%	❌
octopoes/events/events.py	96%	✅
octopoes/events/manager.py	65%	❌
octopoes/models/__init__.py	86%	✅
octopoes/models/datetime.py	66%	❌
octopoes/models/exception.py	83%	✅
octopoes/models/origin.py	70%	❌
octopoes/models/path.py	99%	✅
octopoes/models/types.py	95%	✅
octopoes/models/ooi/certificate.py	96%	✅
octopoes/models/ooi/email_security.py	95%	✅
octopoes/models/ooi/findings.py	94%	✅
octopoes/models/ooi/network.py	97%	✅
octopoes/models/ooi/service.py	91%	✅
octopoes/models/ooi/software.py	71%	❌
octopoes/models/ooi/web.py	81%	✅
octopoes/models/ooi/dns/records.py	95%	✅
octopoes/models/ooi/dns/zone.py	77%	✅
octopoes/repositories/ooi_repository.py	40%	❌
octopoes/repositories/origin_parameter_repository.py	53%	❌
octopoes/repositories/origin_repository.py	53%	❌
octopoes/repositories/scan_profile_repository.py	45%	❌
octopoes/xtdb/client.py	39%	❌
octopoes/xtdb/query_builder.py	69%	❌
octopoes/xtdb/related_field_generator.py	73%	❌
tests/conftest.py	91%	✅

Minimum allowed coverage is 75%

Generated by 🐒 cobertura-action against ec1d09c

[praseodym]

Nice one! Traces show a big improvement, before:

After:

[ammar92]

Simple spelling suggestions, looks good otherwise 👍

[Darwinkel]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make kat.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.

What doesn't work:

• Entering the all organizations crisis room gives an error
  

[Lisser]

Sorry, missed some spots in the naming convention fix.

It's fixed now and tested by myself

[Darwinkel]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make kat.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

• There do not appear to be any regressions.
• Good performance with three organisations with many objects and findings, in the cross-organisation crisis room.