TDL-14888: Respect the start date for all streams #77
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 14 commits
August 20, 2021 17:20
KrisPersonal
pushed a commit
that referenced
this pull request
Oct 14, 2021
* TDL-14621: Add retry logic to requests and TDL-14622: Retry ConnectionResetErrors (#71) * added backoff for certain errors * resolve pylint * updated decorator location * added unittests * added comment * added comments * TDL-14890: Print user friendly error messages (#73) * updated error message when generating auth_stub * made changes according to the comments * updated the code acording to the comments * updated the tap tester image * updated pylint and astroid to latest version * updated the code as, on updating tap tester image it was throwing cascading errors * updated config.yml file * updated the start date for integration tests as per the params * removed scenario as new tap-tester version does not support it * updated start date in the base file test * TDL-14989: Check best practices (#74) * added best practices * resolve pylint * resolve test failure * test: updated the test cases * test: updated some test cases * updated the code as per comments * resolve test case failure * updated the code as per comments * resolve integration test failure * resolve tes case failure * resolve test case failure * add data extension stream in test cases * added data extension incremental stream * test: run bookmark test * test: run bookmark test * test: run bookmark test, debug failing test * test: pylint resolve * test: run all data extenstion stream test in bookmark test * test: updated data extension code * updated the test to run data extension stream * run all tests * added sys.exit and updated pylint and astroid to latest versions * resolve pylint * updated the files as per comments * updated the code * TDL-14889: Keys should be marked automatic and TDL-14891: list_sends stream does not bookmark correctly (#75) * make keys automatic * pylint resolve * add full replication test case * added code change for data extension stream * pylint resolve * added comment * added comments * added comment in base file * updated discovery test and removed full replication test * updated the code * added a comment explaining subscriber and list subscriber syncing * added comments * TDL-14887: Respect field selection (#76) * make keys automatic * pylint resolve * add full replication test case * use transformation * resolve pylint * added code change for data extension stream * pylint resolve * updated test case for data extension * added comment * added comments * added comments and optimized that condition * added code change for tranformation function in base file * pylint resolve * disabled pylint error * test: removed disable pylint code * added comment in base file * updated comment * updated the comment for skipping streams * updated discovery test and removed full replication test * added verification of unique records * updated start date * updated the code * updated the code * added a comment explaining subscriber and list subscriber syncing * added comments * updated comment * made separate files for schemas * resolve pylint * resolve integration test * corrected typo * updated the code * resolved unittest case error * TDL-14888: Respect the start date for all streams (#77) * added code change to use start date when state is not passed * updated code based on comments * resolve test case failure * added data extension stream in tests * add data extension incremental stream * added future date test * updated start date test case * updated future start date test code * updated the code * added comment * added more comments in the code * resolve unittests failure * resolve integration test failure * TDL-14896: Make batch_size apply to the first page and TDL-14895: Use the pagination__list_subscriber_interval_quantity config value correctly (#78) * added code change to respect batch size for 1st page sync * updated code according to comment * resolve pylint error * updated code based on comments * resolve test cases failure * added data extension stream in the test * added batch size for data extension stream * pylint resolve * updated logger messages * pylint resolve * added data extension incremental stream * added assertion for verifying that we did not duplicate any records across pages * updated the code * updated the code * resolve unittest failure * updated comments * added comments * resolve unittests failure * Added comments in the code. (#79) * added comments in the code * resolved typo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
TDL-14888: Respect the start date for all streams
Manual QA steps
Risks
Rollback steps