Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a QueuedFile object #434

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add a QueuedFile object #434

wants to merge 1 commit into from

Conversation

msm-cert
Copy link
Member

Your checklist for this pull request

  • I've read the contributing guideline.
  • I've tested my changes by building and running mquery, and testing changed functionality (if applicable)
  • I've added automated tests for my change (if applicable, optional)
  • I've updated documentation to reflect my change (if applicable)

A smaller version of #432.

This PR creates a "core" (database object) of the index queue that will be managed by ursadb. This will let us queue files for indexing, and index them with available plugins.

This PR doesn't add the "indexer" worker yet, since it's nontrivial and needs some careful design.

@msm-cert
Copy link
Member Author

msm-cert commented Oct 30, 2024

On the other hand, this idea needs some design. It's possible that the feature is too powerful and I should just special-case indexing s3 and local disk (because that's what everyone uses).

Finishing #432 correctly will probably require ursadb changes.

def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table(
"queuedfile",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add something like "indexed" column so we know which files are already indexed?
It would easier the workflow and allow to delete indexed files easily.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants