You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you are running upsert the target table is a temporary table. As the merge doesn't happen until the end this means the state is out of sync with the content of the "real" target table. This can be problematic and lead to what I would call unexpected behavior if a job for any reason doesn't reach its end.
I created this(#96) PR for a possible solution using the pre_state_hook.
The text was updated successfully, but these errors were encountered:
The _handle_max_record_age()
https://github.com/meltano/sdk/blob/6708cb995c68ab6f74d4874dfc8f978c3b054ceb/singer_sdk/target_base.py#L284
Gets called every 5 minutes. It it turn calls drain_all()
target-bigquery/target_bigquery/target.py
which writes to the target table and writes out a state.
If you are running upsert the target table is a temporary table. As the merge doesn't happen until the end this means the state is out of sync with the content of the "real" target table. This can be problematic and lead to what I would call unexpected behavior if a job for any reason doesn't reach its end.
I created this(#96) PR for a possible solution using the pre_state_hook.
The text was updated successfully, but these errors were encountered: