You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#115 introduces a graph-index (for #17) which is dropped/ re-inserted before the observation-pipeline runs. If the ETL process is interrupted while loading observations then it leaves the graph and observation indexes in an inconsistent state. If you try re-running the ETL process then nothing happens as it finds the graph-index to be up to date (even though the observation index is out of date and could include partially-loaded graphs).
To recover from this sort of interruption we need to manually delete the observation and graph indices before restarting, e.g.
It'd be nice if it were a bit more transactional or at least didn't leave the indices in an inconsistent state after interruption - e.g. only update the graph index one doc at a time after all the observations are loaded for that graph. That way even if the observation-pipeline was interrupted mid-graph, it'd redo that graph on the next run (and retain any completed ones).
The text was updated successfully, but these errors were encountered:
#115 introduces a graph-index (for #17) which is dropped/ re-inserted before the observation-pipeline runs. If the ETL process is interrupted while loading observations then it leaves the graph and observation indexes in an inconsistent state. If you try re-running the ETL process then nothing happens as it finds the graph-index to be up to date (even though the observation index is out of date and could include partially-loaded graphs).
To recover from this sort of interruption we need to manually delete the observation and graph indices before restarting, e.g.
It'd be nice if it were a bit more transactional or at least didn't leave the indices in an inconsistent state after interruption - e.g. only update the graph index one doc at a time after all the observations are loaded for that graph. That way even if the observation-pipeline was interrupted mid-graph, it'd redo that graph on the next run (and retain any completed ones).
The text was updated successfully, but these errors were encountered: