Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove local maindb driver #2636

Merged
merged 9 commits into from
Nov 15, 2024
Merged

Remove local maindb driver #2636

merged 9 commits into from
Nov 15, 2024

Conversation

lferran
Copy link
Contributor

@lferran lferran commented Nov 15, 2024

Description

We remove the local implementation of the maindb driver for the following reasons:

  • We have opted in for pg as maindb since long time ago.
  • It has never been production-ready, it has some known bugs and it has been only used for testing so far.
  • Right now it's very easy to spin up a local pg instance and make nucliadb depend on it.

This will allow us to:

  • Remove the catalog implementation based on the index node.
  • Then, we can have shard assignation for resources when ingesting messages coming from processing.
  • The above two points allow us to increase/remove the back-pressure mechanism -- we had it in place because we were assigning resources to shards on resource creation, and that, when the ingestion pipeline was too loaded, could make us place way more resources into a shard than we were capable to handle.

How was this PR tested?

Existing tests should cover us. I'm just removing code

Copy link

codecov bot commented Nov 15, 2024

Codecov Report

Attention: Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 86.51%. Comparing base (8620248) to head (12e7671).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
nucliadb/src/nucliadb/standalone/config.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2636      +/-   ##
==========================================
+ Coverage   86.19%   86.51%   +0.32%     
==========================================
  Files         377      376       -1     
  Lines       23776    23602     -174     
==========================================
- Hits        20493    20419      -74     
+ Misses       3283     3183     -100     
Flag Coverage Δ
nucliadb 74.08% <60.00%> (+0.32%) ⬆️
nucliadb-ingest 28.56% <40.00%> (+0.04%) ⬆️
nucliadb-reader 24.55% <40.00%> (+<0.01%) ⬆️
nucliadb-search 37.75% <60.00%> (+0.13%) ⬆️
nucliadb-standalone 46.84% <40.00%> (+0.21%) ⬆️
nucliadb-train 44.96% <40.00%> (+0.21%) ⬆️
nucliadb-writer 39.30% <40.00%> (+0.15%) ⬆️
nucliadb_dataset 55.45% <ø> (ø)
nucliadb_models 84.85% <ø> (ø)
nucliadb_sdk 80.11% <ø> (ø)
nucliadb_sidecar 89.03% <ø> (ø)
nucliadb_telemetry 86.55% <ø> (ø)
nucliadb_utils 84.31% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@lferran lferran requested a review from a team November 15, 2024 10:28
import pytest
import requests
from pytest_docker_fixtures import images # type: ignore
from pytest_docker_fixtures.containers._base import BaseImage # type: ignore

import nucliadb_sdk

images.settings["postgresql"]["version"] = "11"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think nidx needs v13, maybe we can already use that

@lferran lferran merged commit 4f61b75 into main Nov 15, 2024
46 checks passed
@lferran lferran deleted the remove-local-driver branch November 15, 2024 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants