Skip to content

Commit

Permalink
source-gladly: don't emit a checkpoint with no documents every time
Browse files Browse the repository at this point in the history
Only emit an updated checkpoint when there are no documents if the current cursor is sufficiently
old. Emitting a checkpoint with no documents will trigger an immediate re-invocation of the
incremental task, so it shouldn't be done every time.
  • Loading branch information
williamhbaker committed Mar 5, 2024
1 parent 2248495 commit 17c4bf5
Showing 1 changed file with 10 additions and 0 deletions.
10 changes: 10 additions & 0 deletions source-gladly/source_gladly/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,16 @@ async def fetch_events(
f"not updating cursor since last_ts ({last_ts}) <= log_cursor ({log_cursor})"
)
return

# Only emit an updated checkpoint when there are no documents if the current cursor is
# sufficiently old. Emitting a checkpoint with no documents will trigger an immediate
# re-invocation of this task, so it shouldn't be done every time.
if last_ts - log_cursor < timedelta(hours=6):
log.debug(
f"not updating cursor since updated cursor is less than 6 hours newer than prior cursor (updated cursor: {last_ts} vs prior cursor: {log_cursor})"
)
return

else:
# The Events API has millisecond precision for timestamps, so bump up the log cursor one
# millisecond to not re-fetch the last event seen again on the next round. The assumption
Expand Down

0 comments on commit 17c4bf5

Please sign in to comment.