Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(taps): Allow sort checking to be disabled #730

Conversation

ilkkapeltola
Copy link
Contributor

@ilkkapeltola ilkkapeltola commented Jun 17, 2022

This change adds a new boolean property to core: check_sorted
By default, check_sorted = True, which is the behavior currently when is_sorted = True.
However, there are API's that are sorted, but not by replication_key.

This change allows for an API that is truly ordered, but where the replication_key is not an incremental one.

Closes #729

@edgarrmondragon edgarrmondragon self-requested a review June 17, 2022 13:32
Copy link
Contributor

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ilkkapeltola - thanks for this contribution! 🙏

This looks good to me on first pass but I'll ask @edgarrmondragon to take a more detailed review.

If there are any tests you could add, those would be appreciated, but I think most of this does flow through existing test cases. I think codecov will give info on which code paths are reached or not, if that's helpful.

Also, if can you add info on manual testing you've run, that's always appreciated as well.

@edgarrmondragon edgarrmondragon changed the title feat(taps): Issue 729, allow sort checking to be disabled feat(taps): Allow sort checking to be disabled Jun 17, 2022
@codecov-commenter
Copy link

Codecov Report

Merging #730 (2197749) into main (2166416) will decrease coverage by 1.34%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #730      +/-   ##
==========================================
- Coverage   86.84%   85.49%   -1.35%     
==========================================
  Files          34       34              
  Lines        3383     3386       +3     
==========================================
- Hits         2938     2895      -43     
- Misses        445      491      +46     
Impacted Files Coverage Δ
singer_sdk/helpers/_state.py 77.57% <100.00%> (-1.87%) ⬇️
singer_sdk/streams/core.py 87.17% <100.00%> (-1.75%) ⬇️
singer_sdk/helpers/_catalog.py 75.51% <0.00%> (-16.33%) ⬇️
singer_sdk/helpers/_typing.py 64.34% <0.00%> (-9.31%) ⬇️
singer_sdk/streams/rest.py 88.51% <0.00%> (-4.73%) ⬇️
singer_sdk/authenticators.py 70.17% <0.00%> (-3.51%) ⬇️
singer_sdk/tap_base.py 72.76% <0.00%> (-1.88%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2166416...2197749. Read the comment docs.

Copy link
Collaborator

@edgarrmondragon edgarrmondragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ilkkapeltola This looks good to me! We'll ship this in the next release 😄

@edgarrmondragon edgarrmondragon enabled auto-merge (squash) June 17, 2022 18:21
@edgarrmondragon edgarrmondragon merged commit bf90602 into meltano:main Jun 17, 2022
@ilkkapeltola ilkkapeltola deleted the 729-support-sorted-apis-without-incremental-replication_key branch June 20, 2022 08:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for truly sorted API's where replication_key is not an incremental one
4 participants