Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: orphan chunk state witness pool #10613

Merged
merged 37 commits into from
Feb 23, 2024
Merged

Commits on Feb 21, 2024

  1. Remove witness retrying logic on missing block

    Sometimes when a ChunkStateWitness arrives to be processed, the
    block required to process it isn't available yet, we have to wait
    for it. near#10535 implemented a
    hacky way to do it by retrying the processing every 500ms until the
    required block arrives. This PR will implement a proper solution, so
    let's remove the hacky workaround.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    8dd0c4e View commit details
    Browse the repository at this point in the history
  2. Add verify_chunk_state_witness_signature_in_epoch

    The current function to verify state witness signatures
    takes the epoch_id from the previous block. For orphaned
    state witnesses the previous block isn't available, so
    it's impossible to use this function. Let's add another function
    that takes the epoch_id as an argument instead of fetching it
    from a block. It will allow us to verify signatures of orphaned
    chunk state witnesses.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    456a821 View commit details
    Browse the repository at this point in the history
  3. Add OrphanStateWitnessPool

    Add a struct which keeps a cache of orphaned ChunkStateWitnesses.
    It provides two methods that allow to easily add a ChunkStateWitness
    to the pool, and then when a block arrrives, another method allows
    to easily fetch all orphaned ChunkStateWitnesses that were waiting
    for this block.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    16828b3 View commit details
    Browse the repository at this point in the history
  4. Split process_chunk_state_witness into two parts

    Let's split this function into two parts - one part which
    tries to find the previous block for the given ChunkStateWitness
    and another which processes the witness when the previous block
    is available.
    It will make handling orphaned witnesses easier - once the block
    arrives, we can just call the second function to process it.
    
    In the future it might also make it easier to process witnesses
    before the previous block is saved to the database, which could
    reduce latency.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    d3dc400 View commit details
    Browse the repository at this point in the history
  5. Add possible_epochs_of_height_around_tip

    Add a function which tries to determine in which epoch
    the specified height is located.
    It looks at the previous, current and next epoch around
    the chain Tip and tries to figure out to which one of
    them this height belongs.
    It's not always possible to determine in which epoch
    a given height will be, as the start height of the next
    epoch isn't known in the current epoch, so the function
    returns a set of possible epochs where the height might be.
    
    For example if the tip is in the middle of the current epoch,
    and the height is about epoch_length blocks higher than the tip,
    this function would return [current_epoch_id, next_epoch_id],
    as this height could be in either one of those epochs.
    
    It will later be used to verify the signature of orphaned witnesses.
    To verify the signature we need to know what is the chunk producer
    who should produce a witness at some height, and for that we need
    to know what's the epoch_id at this height.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    5da1fea View commit details
    Browse the repository at this point in the history
  6. Add orphan state witnesses to the orphan pool

    When the previous block isn't available, we can't process the ChunkStateWitness
    and it becomes an orphaned state witness, waiting for the required block to arrive.
    In such situations the witness is partialy validated and then put in the `OrphanStateWitnessPool`.
    It's impossible to fully validate the witness, as the previous block isn't available,
    but we can do some basic checks to curb abusive behavior - check the shard_id, signature,
    size, etc.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    9cb7df9 View commit details
    Browse the repository at this point in the history
  7. Process orphaned witnesses when a new block is accepted

    When a new block is accepted we can process all orphaned
    state witnesses that were waiting for this block.
    Witnesses which became ready are taken out of the pool and processed one by one.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    5a76c7c View commit details
    Browse the repository at this point in the history
  8. Allow to configure orphan pool size from config.js

    There is no ideal size of the witness pool, node operators
    might want to adjust it to achieve optimal performance.
    Let's make this possible by adding a configuration option
    which allows to control the pool size.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    97a0482 View commit details
    Browse the repository at this point in the history
  9. Add metrics to improve observability of OrphanStateWitnessPool

    Add metrics which provide information about witnesses kept in the pool.
    
    Metrics are increased and decreased using a RAII struct to ensure
    that they're always correct, no matter what happens with the pool.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    f9b9c52 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    42af070 View commit details
    Browse the repository at this point in the history
  11. Add TestEnv::get_client_index()

    There was no way to get the index of a client with some AccountId,
    let's add a method that enables it to `TestEnv`
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    6592ace View commit details
    Browse the repository at this point in the history
  12. Add TestEnv::wait_for_chunk_endorsement

    Add a function that allows to wait until some chunk endorsement is sent out.
    
    It will later be used to verify that a validator sends out endorsements
    for an orphaned chunk state witness when its block arrives.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    a29b8df View commit details
    Browse the repository at this point in the history
  13. Expose some items that are needed for the integration tests

    `stateless_validation` is a private module of `near_client`, so it's impossible
    to use items defined there in integration tests. Let's expose some of them to
    make their use in tests possible.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    22a2b15 View commit details
    Browse the repository at this point in the history
  14. Move get_block_producer to TestEnv

    `run_chunk_validation_test` has a very useful function which allows
    to determine a block producer at some offset. I would like to also
    use it in my integration tests.
    Let's add it to `TestEnv` so that other tests can also use it.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    9974f09 View commit details
    Browse the repository at this point in the history
  15. Add TestEnv::get_chunk_producer_at_offset

    Add a function analogous to `get_block_producer_at_offset`, but for chunk producers.
    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    dc19727 View commit details
    Browse the repository at this point in the history
  16. Add integration tests

    jancionear committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    0183075 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. Use ByteSize

    jancionear committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    aedb405 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    198a428 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    49bea32 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    36b7391 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f4cd1e7 View commit details
    Browse the repository at this point in the history
  6. Remove extra newline

    jancionear committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    3b3f8c4 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    db14aa8 View commit details
    Browse the repository at this point in the history
  8. Apply review comments

    jancionear committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    3c045e2 View commit details
    Browse the repository at this point in the history
  9. Use ChunkStateWitness::new_dummy in another test

    There is another test that created a ChunkStateWitness manually.
    Let's use the newly introduced `ChunkStateWitness::new_dummy`
    in this test, it simplifies things.
    jancionear committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    3bcd5c5 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    0b126b8 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    a6858eb View commit details
    Browse the repository at this point in the history
  12. I'm bad at math

    jancionear committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    0edf57a View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. fix log message when a witness is ejected

    In previous commit I added debug spans to avoid
    writing witness_height, ... everywhere. I also
    removed them from the ejection message, but that
    was a mistake. The span contains information about
    the witness that is being added, not the one that
    is being ejected. Because of that the message about
    ejecting a witness would contain invalid witness_height
    and other parameters. Fix by using the correct parameters,
    and names which don't collide with the ones in the span.
    jancionear committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    f61f34c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bb41b37 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    33248ac View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4e6711d View commit details
    Browse the repository at this point in the history
  5. Fix the import of near-o11y

    I imported near-o11y to use the function for initializing
    a tracing-subscriber in epoch manager tests, but I didn't
    import the corresponding nightly packages, so a CI check failed.
    Let's fix this by importing the packages that the CI check
    prosposed.
    jancionear committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    24b25f5 View commit details
    Browse the repository at this point in the history
  6. Fix formatting of epoch-manager/Cargo.toml

    Ran
    python3 scripts/fix_nightly_feature_flags.py fix
    to satisfy the CI
    jancionear committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    58be57b View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    056393d View commit details
    Browse the repository at this point in the history
  8. fix: Don't remove the handler for ChunkEndorsementMessage

    While resolving a merge conflict caused by near#10646,
    I accidentally removed a handler for chunk endorsement messages.
    I didn't mean to do it, I wanted to only modify handlers for chunk state witnesses.
    Let's undo the change and bring the chunk endorsement handler back.
    jancionear committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    5cae3c3 View commit details
    Browse the repository at this point in the history
  9. Revert "fix: Don't remove the handler for ChunkEndorsementMessage"

    This reverts commit 5cae3c3.
    
    It was actually necessary to remove the endorsement message handler,
    it doesn't compile otherwise 🤦
    jancionear committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    6e9b143 View commit details
    Browse the repository at this point in the history