Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try out making a read cache for poolmanager pool modules #8128

Merged
merged 6 commits into from
Apr 27, 2024

Conversation

ValarDragon
Copy link
Member

@ValarDragon ValarDragon commented Apr 23, 2024

Try out making a read cache to avoid extra unmarshals.

For the state machine im pretty confident it will work, it may break the errors for some weird queries to old heights (since it will share this cache, and should really return a miss). But this will only be an edge case for new pools.

We should def test it in prod, but I think this should eliminate this 4.73 out of 1100 seconds of sync time:
image

More importantly i think this will be helpful as a stepping stone to figuring out a more general, re-usable design pattern for us. We could just cache the proto unmarshals, but we would then still pay the key formatting cost. We could cache both key formatting and proto unmarshal in this case.

More general case, we likely do need to cache the two steps differently and use LRU's.

Summary by CodeRabbit

  • New Features

    • Introduced caching for pool module results to enhance performance.
    • Added gas tracking for key-value store operations to improve resource management.
  • Bug Fixes

    • Improved error handling in pool creation tests by including pool ID context.
  • Refactor

    • Transitioned method receivers to pointers for consistency and efficiency in pool management modules.
  • Chores

    • Implemented synchronization mechanisms in pool management to ensure thread safety.
    • Added reset functionality for cache in testing environments to maintain test accuracy.

Copy link
Contributor

Important Notice

This PR modifies an in-repo Go module. It is one of:

  • osmomath
  • osmoutils
  • x/ibc-hooks
  • x/epochs

The dependent Go modules, especially the root one, will have to be
updated to reflect the changes. Failing to do so might cause e2e to fail.

Please follow the instructions below:

  1. Open https://github.com/osmosis-labs/osmosis/actions/workflows/go-mod-auto-bump.yml
  2. Provide the current branch name
  3. On success, confirm if an automated commit corretly updated the go.mod and go.sum files

Please let us know if you need any help.

@ValarDragon ValarDragon added V:state/compatible/backport State machine compatible PR, should be backported A:backport/v24.x backport patches to v24.x branch labels Apr 23, 2024
@github-actions github-actions bot added the C:app-wiring Changes to the app folder label Apr 23, 2024
@ValarDragon ValarDragon changed the title Try out making a read cache Try out making a read cache for poolmanager pool modules Apr 23, 2024
Copy link
Contributor

coderabbitai bot commented Apr 23, 2024

Walkthrough

The recent updates focus on improving efficiency and tracking within the Osmosis project. Key changes include adding caching for GetPoolModule to boost performance, introducing gas tracking for store operations, and updating method receiver types to pointers for consistency. Testing enhancements and synchronization features ensure robustness and concurrency safety.

Changes

Files Change Summary
CHANGELOG.md Added caching for GetPoolModule.
app/apptesting/..., x/poolmanager/keeper.go, x/poolmanager/export_test.go Enhanced caching and added ResetCaches in testing and keeper structures.
osmoutils/store_helper.go, osmoutils/store_helper_test.go Introduced functions for tracking and charging gas in store operations.
x/poolmanager/create_pool.go, x/poolmanager/router.go, x/poolmanager/router_test.go Updated receiver types, added caching, and error handling enhancements in pool management.

Poem

🐇🌟
In the code where data streams,
A rabbit hopped through lines of dreams.
Caches set, and gas was tracked,
With every leap, no speed lacked.
Osmosis flows, more swift, more keen!
🌊💾


Recent Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits Files that changed from the base of the PR and between e220918 and a171092.
Files ignored due to path filters (5)
  • go.mod is excluded by !**/*.mod, !**/*.mod
  • go.sum is excluded by !**/*.sum, !**/*.sum
  • osmoutils/go.mod is excluded by !**/*.mod, !**/*.mod
  • x/epochs/go.mod is excluded by !**/*.mod, !**/*.mod
  • x/ibc-hooks/go.mod is excluded by !**/*.mod, !**/*.mod
Files selected for processing (1)
  • CHANGELOG.md (1 hunks)
Additional comments not posted (1)
CHANGELOG.md (1)

61-61: Ensure the new entry for PR #8128 under "Unreleased" is correctly formatted and consistent with other entries.


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Out of diff range and nitpick comments (2)
CHANGELOG.md (1)

56-56: Ensure consistency in heading levels for better document structure.

x/poolmanager/router_test.go (1)

226-232: > 📝 NOTE

This review was outside the diff hunks and was mapped to the diff hunk with the greatest overlap. Original lines [229-238]

Ensure error messages provide sufficient context for debugging.

Consider enhancing the error message in the s.Require().Error assertion to include more details about the expected and actual outcomes, which can aid in quicker issue resolution during test failures.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Out of diff range and nitpick comments (9)
CHANGELOG.md (9)

Line range hint 1010-1010: Heading levels should only increment by one level at a time. Please adjust the heading level to maintain a logical structure.


Line range hint 1022-1022: Heading levels should only increment by one level at a time. Please adjust the heading level to maintain a logical structure.


Line range hint 1031-1031: Heading levels should only increment by one level at a time. Please adjust the heading level to maintain a logical structure.


Line range hint 1036-1036: Heading levels should only increment by one level at a time. Please adjust the heading level to maintain a logical structure.


Line range hint 1088-1088: Heading levels should only increment by one level at a time. Please adjust the heading level to maintain a logical structure.


Line range hint 349-349: Remove the trailing punctuation in the heading to maintain consistency with other headings.


Line range hint 612-612: Convert the bare URL to a proper link by adding descriptive text.


Line range hint 670-670: Convert the bare URL to a proper link by adding descriptive text.


Line range hint 1251-1251: Ensure there are no empty links in the document. Provide appropriate link text or remove the empty link.

@PaddyMc PaddyMc added the A:backport/v25.x backport patches to v25.x branch label Apr 27, 2024
@ValarDragon ValarDragon merged commit 76d390b into main Apr 27, 2024
1 check passed
@ValarDragon ValarDragon deleted the dev/test_out_read_cahce branch April 27, 2024 18:18
mergify bot pushed a commit that referenced this pull request Apr 27, 2024
* Try out making a read cache

* Make this stored in the keeper to fix tests

* Fix test reset caches

* Auto: update go.mod after push to dev/test_out_read_cahce that modified dependencies locally

* Add changelog

---------

Co-authored-by: github-actions <[email protected]>
(cherry picked from commit 76d390b)

# Conflicts:
#	go.mod
#	go.sum
#	x/ibc-hooks/go.mod
mergify bot pushed a commit that referenced this pull request Apr 27, 2024
* Try out making a read cache

* Make this stored in the keeper to fix tests

* Fix test reset caches

* Auto: update go.mod after push to dev/test_out_read_cahce that modified dependencies locally

* Add changelog

---------

Co-authored-by: github-actions <[email protected]>
(cherry picked from commit 76d390b)
ValarDragon added a commit that referenced this pull request Apr 28, 2024
* Try out making a read cache

* Make this stored in the keeper to fix tests

* Fix test reset caches

* Auto: update go.mod after push to dev/test_out_read_cahce that modified dependencies locally

* Add changelog

---------

Co-authored-by: github-actions <[email protected]>
(cherry picked from commit 76d390b)

Co-authored-by: Dev Ojha <[email protected]>
@github-actions github-actions bot mentioned this pull request May 1, 2024
czarcas7ic added a commit that referenced this pull request May 2, 2024
* chore: add v25.x backport tag to mergify github action (#8151)

* Try out making a read cache for poolmanager pool modules (#8128)

* Try out making a read cache

* Make this stored in the keeper to fix tests

* Fix test reset caches

* Auto: update go.mod after push to dev/test_out_read_cahce that modified dependencies locally

* Add changelog

---------

Co-authored-by: github-actions <[email protected]>

* chore(deps): bump github.com/hashicorp/go-getter from 1.7.1 to 1.7.4 (#8155)

Bumps [github.com/hashicorp/go-getter](https://github.com/hashicorp/go-getter) from 1.7.1 to 1.7.4.
- [Release notes](https://github.com/hashicorp/go-getter/releases)
- [Changelog](https://github.com/hashicorp/go-getter/blob/main/.goreleaser.yml)
- [Commits](hashicorp/go-getter@v1.7.1...v1.7.4)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-getter
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/hashicorp/go-getter in /osmoutils (#8156)

Bumps [github.com/hashicorp/go-getter](https://github.com/hashicorp/go-getter) from 1.7.1 to 1.7.4.
- [Release notes](https://github.com/hashicorp/go-getter/releases)
- [Changelog](https://github.com/hashicorp/go-getter/blob/main/.goreleaser.yml)
- [Commits](hashicorp/go-getter@v1.7.1...v1.7.4)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-getter
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/hashicorp/go-getter in /x/ibc-hooks (#8154)

Bumps [github.com/hashicorp/go-getter](https://github.com/hashicorp/go-getter) from 1.7.1 to 1.7.4.
- [Release notes](https://github.com/hashicorp/go-getter/releases)
- [Changelog](https://github.com/hashicorp/go-getter/blob/main/.goreleaser.yml)
- [Commits](hashicorp/go-getter@v1.7.1...v1.7.4)

---
updated-dependencies:
- dependency-name: github.com/hashicorp/go-getter
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* .5s epoch speedup (#8159)

* Speedup protorev by having route calculation phase not unmarshal pools (#8157)

* Another expected ~1% state machine speedup

* Add changelog

* minor other improvement

* Reduce addr decoding overhead

* chore: Create CI for bumping sdk version via GA (#8162)

* Create CI for updating sdk version

* Fix some comments

* On PR

* Use osmo v0.47.5 as default

* Fix

* Fix 2

* Fix 4

* Fix

* Final

* Final fix

* Fix PR ver

* Final

* Final

* Final

* fix: return err if type not in manager (#8170)

* chore(deps): bump actions/checkout from 3 to 4 (#8173)

Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump actions/setup-go from 3 to 5 (#8174)

Bumps [actions/setup-go](https://github.com/actions/setup-go) from 3 to 5.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](actions/setup-go@v3...v5)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* auto: update Go import paths to v25 on branch main (#8172)

* auto: update Go import paths to v25

* Generated protofile changes

* chore: update sqsdomain to v25

* chore: update osmoutils to latest version

---------

Co-authored-by: PaddyMc <[email protected]>
Co-authored-by: github-actions <[email protected]>
Co-authored-by: PaddyMc <[email protected]>

* chore: don't use hardcoded "uosmo" string (#8145)

* check denoms for gauge creation

* changelog

* extract logic

* use uosmo var

* initial push of test fixes

* fix lint

* tidy

* tidy

* add comments

* add test cases

* chore: update cometbft and cosmos-sdk deps (#8179)

* chore: update cometbft and cosmos-sdk deps

* chore: update comments for cosmos-sdk

* auto: update asset lists (#8180)

Co-authored-by: czarcas7ic <[email protected]>

* Properly create the module account for auction (#8178)

* properly create the module account for auction

* typo

* lint

* preserve name

* updated osmoutils

* chore: consensus param change (#8177)

* consensus param change

* changelog

* err

* Update upgrades.go

---------

Co-authored-by: PaddyMc <[email protected]>

* fix: e2e and remove toolchain (#8191)

* remove toolchain

* remove exclude increment

* script

* remove build tags

* build script

* remove build arg

* fix: e2e and remove toolchain (#8191)

* remove toolchain

* remove exclude increment

* script

* remove build tags

* build script

* remove build arg

(cherry picked from commit 27548cc)

# Conflicts:
#	osmoutils/go.sum
#	x/ibc-hooks/go.sum

* fix conflict

* tidy

* Revert "tidy"

This reverts commit fabe8ea.

* Revert "fix conflict"

This reverts commit 32031fd.

* Revert "Merge branch 'main' into mergify/bp/v25.x/pr-8191"

This reverts commit 708dbdf, reversing
changes made to 29a0917.

* tidy

* run script

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: PaddyMc <[email protected]>
Co-authored-by: Dev Ojha <[email protected]>
Co-authored-by: github-actions <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Matt, Park <[email protected]>
Co-authored-by: Osmosis <[email protected]>
Co-authored-by: PaddyMc <[email protected]>
Co-authored-by: Adam Tucker <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: czarcas7ic <[email protected]>
Co-authored-by: Nicolas Lara <[email protected]>
Co-authored-by: Adam Tucker <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A:backport/v24.x backport patches to v24.x branch A:backport/v25.x backport patches to v25.x branch C:app-wiring Changes to the app folder C:x/epochs C:x/poolmanager V:state/compatible/backport State machine compatible PR, should be backported
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants