Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

command: use external sort for comparison in sync #483

Merged
merged 34 commits into from
Jun 16, 2023
Merged

command: use external sort for comparison in sync #483

merged 34 commits into from
Jun 16, 2023

Commits on Aug 9, 2022

  1. command/storage: move sort logic of sync to lanrat/extsort

    @kucukaslan
    kucukaslan committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    a557522 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into sync-external-sort

    @kucukaslan
    kucukaslan committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    a5e8dfe View commit details
    Browse the repository at this point in the history

  3. reformat Sync methods to use channels instead of slices

    @kucukaslan
    kucukaslan committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    338bde9 View commit details
    Browse the repository at this point in the history

  4. bugfix after refactor

    @kucukaslan
    kucukaslan committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    24241ae View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. specify external sort configuration

    @kucukaslan
    kucukaslan committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    bb56d9b View commit details
    Browse the repository at this point in the history

  2. add 'chunk-size' flag to change chunk size of external sort.

    @kucukaslan
    kucukaslan committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    5643473 View commit details
    Browse the repository at this point in the history

  3. use json instead of xml 

    reduce default chunk size to 100000
    @kucukaslan
    kucukaslan committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    f6fdea5 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2022

  1. force in-memory sort for sync.

    @kucukaslan
    kucukaslan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    6bb085b View commit details
    Browse the repository at this point in the history

  2. omit etag, use format/parse functions

    @kucukaslan
    kucukaslan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    f90564f View commit details
    Browse the repository at this point in the history

  3. improve comments, linting

    @kucukaslan
    kucukaslan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    4c3cbfc View commit details
    Browse the repository at this point in the history

  4. use custom internal sort

    @kucukaslan
    kucukaslan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    c024f5f View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2022

  1. check if the list is already sorted in new internal sort implementat… 

    …ion.
    @kucukaslan
    kucukaslan committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    e15e900 View commit details
    Browse the repository at this point in the history

  2. remove redundant call to ToSlash

    @kucukaslan
    kucukaslan committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    a6ed11d View commit details
    Browse the repository at this point in the history

  3. linting, clarify comments, remove commented out code

    @kucukaslan
    kucukaslan committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    ea12563 View commit details
    Browse the repository at this point in the history

  4. revert the overcompensation

    @kucukaslan
    kucukaslan committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    2812033 View commit details
    Browse the repository at this point in the history

  5. remove commented code

    @kucukaslan
    kucukaslan committed Aug 16, 2022
    Configuration menu
    Copy the full SHA
    8f37dbd View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2022

  1. Merge branch 'master' into sync-external-sort

    @kucukaslan
    kucukaslan committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    e5cc667 View commit details
    Browse the repository at this point in the history

  2. Update CHANGELOG.md

    @kucukaslan
    kucukaslan committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    f841f68 View commit details
    Browse the repository at this point in the history

  3. Update CHANGELOG.md

    @kucukaslan
    kucukaslan committed Aug 23, 2022
    Configuration menu
    Copy the full SHA
    6b64268 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2022

  1. remove chunk-size flag 

    always use external sort by default.
    @kucukaslan
    kucukaslan committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    cdec06d View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into sync-external-sort

    @kucukaslan
    kucukaslan committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    1f8a70f View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2022

  1. fix the bug caused by unconsumed channel 

    The destination only objects are only needed if they're going to be deleted.  So the consumer of onlyDest channel worked if they're going to be deleted.
This naive behaviour caused channel not to accept URLs and block send requests when the channel buffer is filled!

Moral of the story, don't be naive. The fact that you need channel in one case does not mean you need to consume channel only in that case.
    @kucukaslan
    kucukaslan committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    2d397df View commit details
    Browse the repository at this point in the history

  2. correct the comment

    @kucukaslan
    kucukaslan committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    c94fb14 View commit details
    Browse the repository at this point in the history

  3. print external sort errors 

    as an additional benefit, reduce the risk of blocking sends to those channel
    @kucukaslan
    kucukaslan committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    a14ec8c View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'upstream/master' into sync-external-sort

    @kucukaslan
    kucukaslan committed Sep 1, 2022
    Configuration menu
    Copy the full SHA
    ee590ed View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2022

  1. url: gob encoding

    @kucukaslan
    kucukaslan committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    920e539 View commit details
    Browse the repository at this point in the history

  2. storage: use gob encoding for Object

    @kucukaslan
    kucukaslan committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    00ea92a View commit details
    Browse the repository at this point in the history

  3. use RFC3339Nano format to store time instead of the default gob encof… 

    …ding to increase speed

This change reduced the time it takes for 1.000.000 encode/decode operation from 30 seconds to 10 seconds.
    @kucukaslan
    kucukaslan committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    0a2356b View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into sync-external-sort

    @kucukaslan
    kucukaslan authored Sep 15, 2022
    Configuration menu
    Copy the full SHA
    4ac36b3 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2022

  1. command: don't print error if the sync destination is empty

    @kucukaslan
    kucukaslan committed Sep 16, 2022
    Configuration menu
    Copy the full SHA
    171d607 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into sync-external-sort

    @kucukaslan
    kucukaslan authored Sep 16, 2022
    Configuration menu
    Copy the full SHA
    6ab2fd6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into sync-external-sort

    @kucukaslan
    kucukaslan committed Sep 16, 2022
    Configuration menu
    Copy the full SHA
    02d4952 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2022

  1. Merge remote-tracking branch 'upstream/master' into sync-external-sort

    @kucukaslan
    kucukaslan committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    aa67a21 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2023

  1. Merge branch 'master' into sync-external-sort

    @kucukaslan
    kucukaslan authored Feb 3, 2023
    Configuration menu
    Copy the full SHA
    da27b3d View commit details
    Browse the repository at this point in the history