Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Ingestion performance #2433

Closed
johnkerl opened this issue Apr 11, 2024 · 0 comments
Closed

[python] Ingestion performance #2433

johnkerl opened this issue Apr 11, 2024 · 0 comments
Assignees

Comments

@johnkerl
Copy link
Member

johnkerl commented Apr 11, 2024

There are opportunities within the ingestor to try/catch create then open, rather than the other way around, at zero functional change but with improved processing times.

Idea:

  • There are times we want to open something for write if it exists, or create it if it doesn't
    • Sometimes we simply do that: just to get a thing existing at a given URI, whether by creating or opening for write
    • Other times it's an error if the thing already exists (e.g. ingest without ingest_mode="resume")
  • Currently we've been doing a try-open-for-write, and catching a does-not-exist error with trying a create
    • All else being equal this would be as good as doing try-create, catching an already-exists error with trying an open for write
    • However in certain contexts, the if-exists logic in the try-open-for-write case can take unnecessarily long
    • This is particularly relevant for ab nihilo ingests for which the thing not existing is the normal case
    • In either ab nihilo ingests, or append-mode ingests, it turns out to be quite performant to do the try-create first and then fail back to try-open-for-write
    • In summary: we have create and open, and open can be more expensive, so it's always best to try create then open

[sc-44853]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant