You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The ingester workflow runs _append_data() for each each new "section" (entry in a temporary index) of data to the final destination file by first creating a temporary file, copying any existing data to the temporary file, appending the new data and atomically moving the temporary file back to the final destination.
In effect this copies the existing data for every channel time range, which can be many 10s of channels per station, a potential copy-fest!
In some cases, e.g. where rover downloads data at station-day granularity, the append process could be smarter. For example, all data in the same station-day file could be added in a single append operation.
The text was updated successfully, but these errors were encountered:
The ingester workflow runs
_append_data()
for each each new "section" (entry in a temporary index) of data to the final destination file by first creating a temporary file, copying any existing data to the temporary file, appending the new data and atomically moving the temporary file back to the final destination.In effect this copies the existing data for every channel time range, which can be many 10s of channels per station, a potential copy-fest!
In some cases, e.g. where rover downloads data at station-day granularity, the append process could be smarter. For example, all data in the same station-day file could be added in a single append operation.
The text was updated successfully, but these errors were encountered: