TDL-22606 fix none replication_value records sync #122
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RushiT0122
requested review from
dsprayberry and
kethan1122
and removed request for
dsprayberry and
kethan1122
RushiT0122
changed the title
kethan1122
reviewed
Apr 5, 2023
kethan1122
approved these changes
Apr 5, 2023
dsprayberry
changed the title
dsprayberry
requested changes
Apr 5, 2023
dsprayberry
reviewed
Apr 5, 2023
tap_pendo/streams.py
Outdated
| if isinstance(self, Accounts): | ||
| return record['metadata']['auto']['lastupdated'] | ||
|
|
||
| decamalized_replication_key = humps.decamelize(self.replication_key) |
There was a problem hiding this comment.
Typo in "decamalized" should be "decamelized..."
dsprayberry
requested changes
Apr 5, 2023
tap_pendo/streams.py
Outdated
| return record['metadata']['auto']['lastupdated'] | ||
|
|
||
| decamalized_replication_key = humps.decamelize(self.replication_key) | ||
| return record.get(decamalized_replication_key) |
There was a problem hiding this comment.
Typo in "decamalized" should be "decamelized..."
tap_pendo/streams.py
Outdated
| @@ -549,20 +550,41 @@ def set_request_body_filters(self, body, start_time, records=None): | |||
|
|
|||
| return body | |||
|
|
|||
| def get_replication_value(self, record): | |||
There was a problem hiding this comment.
I think we should rename this to be a little more concise, eg get_replication_key_value
tap_pendo/streams.py
Outdated
| @@ -549,20 +550,41 @@ def set_request_body_filters(self, body, start_time, records=None): | |||
|
|
|||
| return body | |||
|
|
|||
| def get_replication_value(self, record): | |||
| """Return the replication value of """ | |||
There was a problem hiding this comment.
This seems like an incomplete thought; was there more to this docstring?
tap_pendo/streams.py
Outdated
| @@ -149,6 +149,7 @@ class Stream(): | |||
| def __init__(self, config=None): | |||
| self.config = config | |||
| self.record_limit = self.get_default_record_limit() | |||
| self.none_replication_value_records = [] | |||
There was a problem hiding this comment.
I think we can rename this to make it slightly clearer; something like empty_replication_key_records' or empty_rk_records' if we want to shorten it
tap_pendo/streams.py
Outdated
| decamalized_replication_key = humps.decamelize(self.replication_key) | ||
| return record.get(decamalized_replication_key) | ||
|
|
||
| def remove_none_replication_value_records(self, records): |
There was a problem hiding this comment.
I think we should rename this to align with the change requested above for line 152. something like remove_empty_rk_records
|
@dsprayberry fixed the comments, please re-review. |
dsprayberry
approved these changes
Apr 5, 2023
RushiT0122
force-pushed
the
patch-accounts-replication-key-issue
branch
from
92d4e4f to
7a62090
Compare
rdeshmukh15
approved these changes
Apr 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
Fix TypeError observed while syncing accounts records (TDL-22606)
There were two scenarios which could cause TypeError which we fixed,
Also, refactored code to avoid duplication code and have generic implementation.
Manual QA steps
record_limitvalue.Risks
Rollback steps