Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingester: existing data copy for each segment added, could be optimized #53

Open
chad-earthscope opened this issue Nov 14, 2018 · 0 comments
Labels
enhancement New feature or request

Comments

@chad-earthscope
Copy link
Contributor

The ingester workflow runs _append_data() for each each new "section" (entry in a temporary index) of data to the final destination file by first creating a temporary file, copying any existing data to the temporary file, appending the new data and atomically moving the temporary file back to the final destination.

In effect this copies the existing data for every channel time range, which can be many 10s of channels per station, a potential copy-fest!

In some cases, e.g. where rover downloads data at station-day granularity, the append process could be smarter. For example, all data in the same station-day file could be added in a single append operation.

@chad-earthscope chad-earthscope added the enhancement New feature or request label Nov 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant