Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flink: Implement data statistics coordinator to aggregate data statistics from operator subtasks #7360

Merged
merged 27 commits into from
Sep 25, 2023
Merged

Flink: Implement data statistics coordinator to aggregate data statistics from operator subtasks #7360

merged 27 commits into from
Sep 25, 2023

Commits on May 16, 2023

  1. Flink: Implement data statistics coordinator to aggregate data statis… 

    …tics from operator subtasks
    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    049f5f7 View commit details
    Browse the repository at this point in the history

  2. fix CICD error

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    f0f84d0 View commit details
    Browse the repository at this point in the history

  3. handle comments to add javadoc for all classes and rename AggregateDa… 

    …taStatistics to GlobalStatisticsAggregator
    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    e550396 View commit details
    Browse the repository at this point in the history

  4. rename lastCompleteAggregator to lastCompletedAggregator

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    c0800be View commit details
    Browse the repository at this point in the history

  5. log msg minor change

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    8f9ec54 View commit details
    Browse the repository at this point in the history

  6. log message minor change

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    61b21e2 View commit details
    Browse the repository at this point in the history

  7. update EXPECTED_DATA_STATISTICS_RECEIVED_PERCENTAGE from 0.8 to 80 an… 

    …d remove start check in coordinator close function
    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    9a3f598 View commit details
    Browse the repository at this point in the history

  8. move executor close function to context

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    c6fe1b4 View commit details
    Browse the repository at this point in the history

  9. remove coordinator context

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    3b441e7 View commit details
    Browse the repository at this point in the history

  10. remove unused comments

    gang_ye committed May 16, 2023
    Configuration menu
    Copy the full SHA
    ed0cb09 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2023

  1. use RowData as the type to collect data statistics

    gang_ye committed May 17, 2023
    Configuration menu
    Copy the full SHA
    9e6a672 View commit details
    Browse the repository at this point in the history

  2. fix style in coordinator

    gang_ye committed May 17, 2023
    Configuration menu
    Copy the full SHA
    dfc0a2f View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. fix unit test to only send bytes in DataStatisticsEvent

    gang_ye committed May 23, 2023
    Configuration menu
    Copy the full SHA
    7a068b9 View commit details
    Browse the repository at this point in the history

  2. fix style

    gang_ye committed May 23, 2023
    Configuration menu
    Copy the full SHA
    fc36623 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2023

  1. fix CoordinatorProvider unit test

    gang_ye committed May 24, 2023
    Configuration menu
    Copy the full SHA
    333bd8f View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. Remove final and update function receiveDataStatisticEventAndCheckCom… 

    …pletion in class GlobalStatisticsAggregatorTracker
    gang_ye committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    ad0203e View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. Create DataStatisticsUtil to serialize and deserialize statistics. Ad… 

    …d operator name to logging
    gang_ye committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    86ca09c View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. handle comments to update log message and unit tests

    gang_ye committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    de9f629 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. remove subtask set in GlobalStatistics and use GlobalStatisticsTracke… 

    …r to track the processed subtasks
    gang_ye committed Aug 17, 2023
    Configuration menu
    Copy the full SHA
    9ce20c5 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2023

  1. fix style to use Set.newHashSet to create set

    gang_ye committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    e5b6dc6 View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2023

  1. update comments log messages

    gang_ye committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    baf838e View commit details
    Browse the repository at this point in the history

  2. rename GlobalStatistics GlobalStatisticsTracker to AggregatedStatisti… 

    …cs and AggregatedStatisticsTracker
    gang_ye committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    6107ac3 View commit details
    Browse the repository at this point in the history

  3. fix style

    gang_ye committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    7bdb2b0 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2023

  1. update log message and comments

    gang_ye committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    02f3df1 View commit details
    Browse the repository at this point in the history

  2. delete GlobalStatisticsTracker

    gang_ye committed Sep 22, 2023
    Configuration menu
    Copy the full SHA
    aab164f View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2023

  1. use assertj in unit test

    gang_ye committed Sep 23, 2023
    Configuration menu
    Copy the full SHA
    8f6cc7a View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2023

  1. handle comments to update log message

    gang_ye committed Sep 24, 2023
    Configuration menu
    Copy the full SHA
    bfc9e05 View commit details
    Browse the repository at this point in the history