Skip to content

Commit

Permalink
source-oracle-flashback: filter out already-processed SCNs
Browse files Browse the repository at this point in the history
VERSIONS BETWEEN SCN results in any valid rows for the specified SCN range, and rows
which have "VERSIONS_STARTSCN < rangeBegin && VERSIONS_ENDSCN >= rangeBegin" are also considered
in our logic however we only consider VERSIONS_STARTSCN as the transaction identifying checkpoint
that's why we need the extra "VERSIONS_STARTSCN >= {{cursor}}" to avoid results where VERSIONS_STARTSCN
is smaller than the latest checkpoint we have emitted.
  • Loading branch information
mdibaiee committed Jan 2, 2025
1 parent 8fad862 commit 18bec33
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions source-oracle-flashback/source_oracle_flashback/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -377,13 +377,19 @@ def cast_column(c: OracleColumn) -> str:
ORDER BY ROWID ASC
FETCH FIRST {{ chunk_size }} ROWS ONLY
""",
# VERSIONS BETWEEN SCN results in any valid rows for the specified SCN range, and rows
# which have "VERSIONS_STARTSCN < rangeBegin && VERSIONS_ENDSCN >= rangeBegin" are also considered
# in our logic however we only consider VERSIONS_STARTSCN as the transaction identifying checkpoint
# that's why we need the extra "VERSIONS_STARTSCN >= {{cursor}}" to avoid results where VERSIONS_STARTSCN
# is smaller than the latest checkpoint we have emitted
'inc': """
SELECT /*+parallel */ VERSIONS_STARTSCN, VERSIONS_OPERATION, ROWID, {% for c in table.columns -%}
{%- if not loop.first %}, {% endif -%}
{{ c | cast }}
{%- endfor %} FROM {{ table.quoted_owner }}.{{ table.quoted_table_name }}
VERSIONS BETWEEN SCN {{ cursor }} AND MAXVALUE
WHERE VERSIONS_STARTSCN IS NOT NULL
AND VERSIONS_STARTSCN >= {{ cursor }}
ORDER BY VERSIONS_STARTSCN ASC
"""
}))
Expand Down

0 comments on commit 18bec33

Please sign in to comment.