Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add PER_QUERY_MEMORY_LIMIT spilling strategy #15898

Merged
merged 5 commits into from
Apr 23, 2021
Merged

Add PER_QUERY_MEMORY_LIMIT spilling strategy #15898

merged 5 commits into from
Apr 23, 2021

Commits on Apr 21, 2021

  1. Refactor MemoryPoolListener 

    Simplify to a functional interface and add queryId and memoryReservation
arg in preparation for per-query spill strategy.
    @rschlussel
    rschlussel committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    f660030 View commit details
    Browse the repository at this point in the history

  2. Refactor TaskRevocableMemoryListener 

    Simplify to functional interface, remove unused parameter, and make
clearer that it's only called on revocable memory reservation.
    @rschlussel
    rschlussel committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    0afc1f2 View commit details
    Browse the repository at this point in the history

  3. Move getMemoryAlreadyBeingRevoked to utility class 

    so that it can be used by the PerQueryMemoryRevokingScheduler
    @rschlussel
    rschlussel committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    a3d6a86 View commit details
    Browse the repository at this point in the history

  4. Add per-query limit based spilling strategy 

    Add spilling strategy that will spill whenever the user + system +
revocabble memory for a query on a node exceeds the
maxTotalMemoryPerNode. This allows queries to have consistent behavior
regardless of the cluster load, while also ensuring that queries don't
use more memory than the cluster can support.
    @rschlussel
    rschlussel committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    e5df45f View commit details
    Browse the repository at this point in the history

  5. Catch Exception in MemoryRevokingSchedulers 

    Catch Exception instead of Throwable to make sure Errors that should be
fatal get propagated (e.g. OutOfMemoryError, etc.)
    @rschlussel
    rschlussel committed Apr 21, 2021
    Configuration menu
    Copy the full SHA
    0d5df77 View commit details
    Browse the repository at this point in the history