Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server/v2) Add prune cmd to serverv2 #20736

Merged
merged 80 commits into from
Jul 23, 2024
Merged

feat(server/v2) Add prune cmd to serverv2 #20736

merged 80 commits into from
Jul 23, 2024

Conversation

hieuvubk
Copy link
Contributor

@hieuvubk hieuvubk commented Jun 20, 2024

Description

Closes: #20511, #19674

  • Moved AppI, AppCreator, CLIConfig from server/v2 to core/server
  • Write & read store config
  • Add prune cmd
  • Added some prune helper to store/v2
  • Added Prune method in store/v2.Store interface

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title, you can find examples of the prefixes below:
  • confirmed ! in the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • reviewed "Files changed" and left comments if necessary
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • updated the relevant documentation or specification, including comments for documenting Go code
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

Please see Pull Request Reviewer section in the contributing guide for more information on how to review a pull request.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

Summary by CodeRabbit

Summary by CodeRabbit

  • New Features

    • Introduced a command to prune app history states, allowing users to manage storage more effectively.
    • Added a default configuration setup for easier application management.
  • Improvements

    • Enhanced pruning options with new strategies and validation rules for better configuration management.
    • Streamlined configuration management by consolidating related settings into a single structure.
  • Dependencies

    • Updated multiple dependencies to enhance configuration management and overall functionality of the application.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Copy link
Contributor

coderabbitai bot commented Jun 20, 2024

Walkthrough

Walkthrough

The primary change introduces new functionality to the store package, adding CLI pruning commands for managing app history states. The PrunesCmd() command supports various pruning strategies, including retaining recent states, archiving, or custom configurations. Additionally, the integration of spf13/cobra and spf13/viper enhances command-line operations and configuration management. Supporting modifications across other files ensure that these new commands operate seamlessly within the application.

Changes

File Change Summary
server/v2/store/commands.go Added PrunesCmd() for pruning app history states based on various strategies and custom options.
server/v2/store/config.go Introduced DefaultConfig function and Config struct for comprehensive store configuration management.
store/v2/options.go Enhanced PruningOption struct with new tags, constants, validation errors, and functions for creating and validating options.
store/v2/root/factory.go Refactored FactoryOptions struct to utilize new Options struct for store configuration.
runtime/v2/go.mod Added several indirect dependencies related to configuration and logging.
server/v2/go.mod Updated replace and require directives for cosmossdk.io/store/v2 and other indirect dependencies.
simapp/v2/go.mod Modified versioning for dependencies, including updates to cosmossdk.io/server/v2 and indirect status for cosmossdk.io/store/v2.
simapp/v2/simdv2/cmd/commands.go Updated serverv2.AddCommands to include store.New[T](), enhancing the command integration.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant CLI
    participant StoreComponent
    participant PruningHandler
    participant Database

    User->>CLI: Execute PrunesCmd()
    CLI->>StoreComponent: Initialize PrunesCmd
    StoreComponent->>PruningHandler: Parse options and prepare for pruning
    PruningHandler->>Database: Retrieve latest height
    PruningHandler->>Database: Calculate pruning heights
    PruningHandler->>Database: Perform pruning operations
    Database-->>PruningHandler: Return pruning results
    PruningHandler-->>StoreComponent: Return success/failure
    StoreComponent-->>CLI: Command execution result
    CLI-->>User: Display results
Loading

Assessment against linked issues

Objective Addressed Explanation
Implementation of CLI pruning commands in simapp/v2 (#20511)
Ensure integration with spf13/cobra and spf13/viper for command-line operations and configuration

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

server/v2/flags.go Outdated Show resolved Hide resolved
@hieuvubk hieuvubk marked this pull request as ready for review June 20, 2024 12:16
@hieuvubk hieuvubk requested a review from a team as a code owner June 20, 2024 12:16
Copy link
Contributor

@hieuvubk your pull request is missing a changelog!

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Outside diff range and nitpick comments (1)
server/v2/store/config.go (1)

17-21: Consider adding comments to the Config struct fields to enhance code readability and maintainability.

Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between ac77fc1 and 3ae993a.

Files ignored due to path filters (1)
  • server/v2/store/go.sum is excluded by !**/*.sum
Files selected for processing (12)
  • server/v2/cometbft/server.go (1 hunks)
  • server/v2/commands.go (1 hunks)
  • server/v2/server.go (2 hunks)
  • server/v2/store/commands.go (1 hunks)
  • server/v2/store/config.go (1 hunks)
  • server/v2/store/flags.go (1 hunks)
  • server/v2/store/go.mod (1 hunks)
  • server/v2/store/server.go (1 hunks)
  • simapp/v2/go.mod (2 hunks)
  • simapp/v2/simdv2/cmd/commands.go (3 hunks)
  • store/v2/root/store.go (1 hunks)
  • store/v2/store.go (1 hunks)
Files skipped from review due to trivial changes (3)
  • server/v2/store/flags.go
  • server/v2/store/go.mod
  • simapp/v2/go.mod
Additional context used
Path-based instructions (9)
server/v2/store/config.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

server/v2/store/server.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

server/v2/commands.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

store/v2/store.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

server/v2/store/commands.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

server/v2/server.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

simapp/v2/simdv2/cmd/commands.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

server/v2/cometbft/server.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

store/v2/root/store.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

Additional comments not posted (12)
server/v2/store/config.go (1)

7-13: Ensure default configuration values align with system requirements.

Verification successful

The default values for AppDBBackend, PruningKeepRecent, and PruningInterval in the DefaultConfig function are consistent with their usage across the codebase. The default values are set to empty or zero, which aligns with the system requirements.

  • AppDBBackend is set to an empty string, which is consistent.
  • PruningKeepRecent is set to 0, which is consistent.
  • PruningInterval is set to 0, which is consistent.

No discrepancies were found in the default configuration values.

Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify the default configuration values align with system requirements.

# Test: Check if the default values are consistent across similar configurations.
rg --type go "PruningOptionDefault"

Length of output: 1243



Script:

#!/bin/bash
# Description: Verify the default configuration values align with system requirements.

# Test: Check if the default values for `AppDBBackend`, `PruningKeepRecent`, and `PruningInterval` are consistent across similar configurations.
rg --type go "AppDBBackend" 
rg --type go "PruningKeepRecent" 
rg --type go "PruningInterval"

Length of output: 10845

server/v2/store/server.go (4)

14-16: The StoreComponent struct is well-defined. Ensure the config field is properly managed throughout the component's lifecycle.


18-20: The New method correctly initializes a StoreComponent. Consider adding error handling if future configurations require it.


46-52: The Config method effectively handles null configurations by returning a default configuration. This is a good fallback mechanism.


22-24: The Init method currently always succeeds. Verify that this is the intended behavior, especially in error scenarios.

server/v2/commands.go (1)

73-73: The modification to use newApp in CLICommands aligns with the changes in the server interfaces. Ensure all dependent components are updated accordingly.

Verification successful

The verification confirms that all instances of CLICommands have been updated to align with the new signature using newApp.

  • server/v2/server.go at line 121
  • server/v2/store/server.go at line 38
  • server/v2/cometbft/server.go at line 218

All dependent components are correctly updated.

Scripts executed

The following scripts were executed for the analysis:


Script:

#!/bin/bash
# Description: Verify all dependent components are updated to use the new `CLICommands` signature.

# Test: Search for outdated uses of `CLICommands`.
rg --type go "CLICommands" --context 5

Length of output: 3250



Script:

#!/bin/bash
# Description: Verify that all instances of `CLICommands` are updated to use the new `CLICommands` signature with `newApp`.

# Verify `CLICommands` implementation in `server/v2/server.go`
ast-grep --lang go --pattern $'func (s *Server) CLICommands(appCreator AppCreator[transaction.Tx]) CLIConfig {
  $$$
}' server/v2/server.go

# Verify `CLICommands` implementation in `server/v2/store/server.go`
ast-grep --lang go --pattern $'func (s StoreComponent) CLICommands(appCreator serverv2.AppCreator[transaction.Tx]) serverv2.CLIConfig {
  $$$
}' server/v2/store/server.go

# Verify `CLICommands` implementation in `server/v2/cometbft/server.go`
ast-grep --lang go --pattern $'func (s *CometBFTServer[T]) CLICommands(_ serverv2.AppCreator[T]) serverv2.CLIConfig {
  $$$
}' server/v2/cometbft/server.go

Length of output: 2552

store/v2/store.go (1)

65-65: The addition of the Prune method to the RootStore interface is crucial for implementing pruning functionality. Ensure it is correctly implemented across all concrete store implementations.

server/v2/store/commands.go (2)

21-101: The implementation of the PrunesCmd function looks solid. It correctly sets up a new Cobra command, handles command-line arguments, and performs the pruning operation. However, consider adding more detailed error messages to improve troubleshooting and user experience.


103-125: The getPruningOptionsFromFlags function is well-implemented with a clear handling of different pruning strategies. However, ensure that all possible error paths are adequately tested to prevent runtime issues.

simapp/v2/simdv2/cmd/commands.go (1)

88-88: The addition of the store.New() to the root command initialization is correctly implemented. Ensure that all dependencies and configurations required by the store component are properly set up before this point.

server/v2/cometbft/server.go (1)

218-218: The CLICommands method is correctly implemented to include various necessary commands. Consider adding documentation or examples on how to use these commands for better user guidance.

store/v2/root/store.go (1)

434-436: The implementation of the Prune method is straightforward and correctly delegates to the pruningManager. However, ensure that pruningManager is always properly initialized before this method is called to prevent nil pointer dereferences.

server/v2/server.go Outdated Show resolved Hide resolved
server/v2/store/server.go Outdated Show resolved Hide resolved
server/v2/store/server.go Outdated Show resolved Hide resolved
// v.Set("store.options.sc-pruning-option.keep-recent", keepRecent) // entry that read from app.toml
// v.Set("store.options.ss-pruning-option.keep-recent", keepRecent)

err = overrideKeepRecent(filepath.Join(rootDir, "config"), keepRecent)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have to do a hacky way cause spf13/viper#1106.
If cmd has keep-recent flag, overwrite app.toml then read it again

}

func overrideKeepRecent(configPath string, keepRecent uint64) error {
bz, err := os.ReadFile(filepath.Join(configPath, "app.toml"))

Check failure

Code scanning / gosec

Potential file inclusion via variable Error

Potential file inclusion via variable
@hieuvubk hieuvubk marked this pull request as ready for review July 22, 2024 04:33
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Outside diff range, codebase verification and nitpick comments (1)
store/v2/options.go (1)

Line range hint 1-1: Use constants for error messages.

The error messages are clear and provide actionable feedback, which is excellent for debugging and user experience. However, consider using constants for the minimum values in the error messages to maintain consistency and ease future modifications.

+ const minPruningInterval = 10
+ const minKeepRecent = 2

- ErrPruningIntervalTooSmall = fmt.Errorf("'pruning-interval' must not be less than %d. For the most aggressive pruning, select pruning = \"everything\"", pruneEverythingInterval)
+ ErrPruningIntervalTooSmall = fmt.Errorf("'pruning-interval' must not be less than %d. For the most aggressive pruning, select pruning = \"everything\"", minPruningInterval)

- ErrPruningKeepRecentTooSmall = fmt.Errorf("'pruning-keep-recent' must not be less than %d. For the most aggressive pruning, select pruning = \"everything\"", pruneEverythingKeepRecent)
+ ErrPruningKeepRecentTooSmall = fmt.Errorf("'pruning-keep-recent' must not be less than %d. For the most aggressive pruning, select pruning = \"everything\"", minKeepRecent)
Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between a7a38ed and 237ef66.

Files ignored due to path filters (3)
  • runtime/v2/go.sum is excluded by !**/*.sum
  • server/v2/go.sum is excluded by !**/*.sum
  • store/v2/go.sum is excluded by !**/*.sum
Files selected for processing (15)
  • runtime/v2/builder.go (4 hunks)
  • runtime/v2/go.mod (5 hunks)
  • runtime/v2/module.go (4 hunks)
  • runtime/v2/types.go (1 hunks)
  • server/v2/go.mod (4 hunks)
  • server/v2/store/commands.go (1 hunks)
  • server/v2/store/config.go (1 hunks)
  • server/v2/store/flags.go (1 hunks)
  • server/v2/store/server.go (1 hunks)
  • simapp/v2/app_di.go (5 hunks)
  • simapp/v2/go.mod (1 hunks)
  • simapp/v2/simdv2/cmd/commands.go (2 hunks)
  • store/v2/go.mod (3 hunks)
  • store/v2/options.go (1 hunks)
  • store/v2/root/factory.go (5 hunks)
Files skipped from review due to trivial changes (1)
  • runtime/v2/types.go
Files skipped from review as they are similar to previous changes (8)
  • runtime/v2/go.mod
  • server/v2/go.mod
  • server/v2/store/config.go
  • server/v2/store/flags.go
  • server/v2/store/server.go
  • simapp/v2/simdv2/cmd/commands.go
  • store/v2/go.mod
  • store/v2/root/factory.go
Additional context used
Path-based instructions (5)
store/v2/options.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

server/v2/store/commands.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

runtime/v2/builder.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

simapp/v2/app_di.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

runtime/v2/module.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

GitHub Check: gosec
server/v2/store/commands.go

[failure] 130-130: Potential file inclusion via variable
Potential file inclusion via variable

Additional comments not posted (20)
store/v2/options.go (3)

Line range hint 1-1: LGTM!

The addition of constants for various pruning strategies and associated parameters is well-defined and improves code readability and maintainability.


Line range hint 1-1: LGTM!

The new functions NewPruningOptionFromString and Validate are well-designed and improve the flexibility and validation of pruning options.


Line range hint 1-1: LGTM!

The modifications to NewPruningOption and the addition of NewPruningOptionWithCustom enhance the functionality of pruning options.

server/v2/store/commands.go (4)

19-83: LGTM!

The PrunesCmd function is well-structured and provides comprehensive CLI documentation for the pruning command. It correctly binds command flags and handles errors appropriately. The use of cobra.RangeArgs ensures that the command does not accept more than one argument, which is a good practice for CLI design.


86-127: LGTM! But monitor for a future fix in Viper.

The createRootStore function is well-designed and handles the creation of the root store and management of pruning options. However, it includes a hack to override the keep-recent value in the app.toml file due to a limitation in Viper. Consider monitoring for a future fix in Viper to remove this workaround.


3-17: LGTM!

The import statements are necessary for the functionality of the file.


1-2: LGTM!

The package declaration is correct and aligns with the file's purpose.

runtime/v2/builder.go (5)

29-29: LGTM!

The addition of the viper field to the AppBuilder struct enhances configuration management.


124-138: LGTM!

The updates to the Build method improve the functionality and structure of the method by including logic for initializing a new database instance and creating store options.


Line range hint 3-19: LGTM!

The import statements are necessary for the functionality of the file.


Line range hint 1-2: LGTM!

The package declaration is correct and aligns with the file's purpose.


Line range hint 1-1: LGTM!

The AppBuilderOption functions are well-designed and enhance the flexibility of the AppBuilder.

simapp/v2/app_di.go (3)

Line range hint 9-9:
LGTM! Import added correctly.

The import of viper is necessary for the new configuration management approach.


Line range hint 154-154:
LGTM! Struct modified correctly.

The addition of the Viper field in the AppInputs struct is consistent with the new configuration management approach.


Line range hint 166-167:
LGTM! Function updated correctly.

The conditional check and assignment of the Viper instance in the SetupAppBuilder function are necessary and correctly implemented.

runtime/v2/module.go (3)

9-9: LGTM! Import added correctly.

The import of viper is necessary for the new configuration management approach.


154-154: LGTM! Struct modified correctly.

The addition of the Viper field in the AppInputs struct is consistent with the new configuration management approach.


166-167: LGTM! Function updated correctly.

The conditional check and assignment of the Viper instance in the SetupAppBuilder function are necessary and correctly implemented.

simapp/v2/go.mod (2)

13-13: LGTM! Version updated correctly.

The update to a specific timestamped version for cosmossdk.io/server/v2 is necessary for stability and clarity.


15-15: LGTM! Dependency marked as indirect correctly.

Marking cosmossdk.io/store/v2 as indirect is appropriate if it is no longer directly used but still required by other packages.

Comment on lines 23 to 27
KeepRecent uint64 `mapstructure:"keep-recent" toml:"keep-recent"`

// Interval sets the number of how often to prune.
// If set to 0, no pruning will be done.
Interval uint64
Interval uint64 `mapstructure:"interval" toml:"interval"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor suggestion: Consistent struct field tagging.

The struct tags for KeepRecent and Interval fields have been added for mapstructure and toml. This is good for serialization and deserialization consistency. However, consider adding JSON tags as well to maintain a uniform approach across different serialization formats.

- KeepRecent uint64 `mapstructure:"keep-recent" toml:"keep-recent"`
- Interval uint64   `mapstructure:"interval" toml:"interval"`
+ KeepRecent uint64 `json:"keepRecent" mapstructure:"keep-recent" toml:"keep-recent"`
+ Interval uint64   `json:"interval" mapstructure:"interval" toml:"interval"`
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
KeepRecent uint64 `mapstructure:"keep-recent" toml:"keep-recent"`
// Interval sets the number of how often to prune.
// If set to 0, no pruning will be done.
Interval uint64
Interval uint64 `mapstructure:"interval" toml:"interval"`
KeepRecent uint64 `json:"keepRecent" mapstructure:"keep-recent" toml:"keep-recent"`
// Interval sets the number of how often to prune.
// If set to 0, no pruning will be done.
Interval uint64 `json:"interval" mapstructure:"interval" toml:"interval"`

Comment on lines +129 to +144
func overrideKeepRecent(configPath string, keepRecent uint64) error {
bz, err := os.ReadFile(filepath.Join(configPath, "app.toml"))
if err != nil {
return err
}
lines := strings.Split(string(bz), "\n")

for i, line := range lines {
if strings.Contains(line, "keep-recent") {
lines[i] = fmt.Sprintf("keep-recent = %d", keepRecent)
}
}
output := strings.Join(lines, "\n")

return os.WriteFile(filepath.Join(configPath, "app.toml"), []byte(output), 0o600)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential file inclusion vulnerability.

The overrideKeepRecent function modifies the app.toml file to set the keep-recent value. However, it includes a potential file inclusion vulnerability. Ensure that the file path is properly sanitized and validated before reading or writing to the file.

- bz, err := os.ReadFile(filepath.Join(configPath, "app.toml"))
+ configFilePath := filepath.Join(configPath, "app.toml")
+ if !filepath.IsAbs(configFilePath) || !strings.HasPrefix(configFilePath, configPath) {
+     return fmt.Errorf("invalid file path: %s", configFilePath)
+ }
+ bz, err := os.ReadFile(configFilePath)
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
func overrideKeepRecent(configPath string, keepRecent uint64) error {
bz, err := os.ReadFile(filepath.Join(configPath, "app.toml"))
if err != nil {
return err
}
lines := strings.Split(string(bz), "\n")
for i, line := range lines {
if strings.Contains(line, "keep-recent") {
lines[i] = fmt.Sprintf("keep-recent = %d", keepRecent)
}
}
output := strings.Join(lines, "\n")
return os.WriteFile(filepath.Join(configPath, "app.toml"), []byte(output), 0o600)
}
func overrideKeepRecent(configPath string, keepRecent uint64) error {
configFilePath := filepath.Join(configPath, "app.toml")
if !filepath.IsAbs(configFilePath) || !strings.HasPrefix(configFilePath, configPath) {
return fmt.Errorf("invalid file path: %s", configFilePath)
}
bz, err := os.ReadFile(configFilePath)
if err != nil {
return err
}
lines := strings.Split(string(bz), "\n")
for i, line := range lines {
if strings.Contains(line, "keep-recent") {
lines[i] = fmt.Sprintf("keep-recent = %d", keepRecent)
}
}
output := strings.Join(lines, "\n")
return os.WriteFile(filepath.Join(configPath, "app.toml"), []byte(output), 0o600)
}
Tools
GitHub Check: gosec

[failure] 130-130: Potential file inclusion via variable
Potential file inclusion via variable

Copy link
Member

@julienrbrt julienrbrt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly lgtm! let's remove viper dep from store/v2

@@ -116,6 +121,22 @@ func (a *AppBuilder[T]) Build(opts ...AppBuilderOption[T]) (*App[T], error) {

a.app.stf = stf

v := a.viper
home := v.GetString(FlagHome)
scRawDb, err := db.NewGoLevelDB("application", filepath.Join(home, "data"), nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so store/v2 is goleveldb only @kocubinski?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can handle multiple dbs by db-backend in app.toml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's for v1, but maybe v2 doesn't support multiple backend? Wanted to confirm, otherwise this being hardcoded in runtime isn't great (but easily solvable with an appbuilder options). Let's wait for an answer

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

store v2 has support for pebbledb as well

simapp/v2/app_di.go Show resolved Hide resolved
store/v2/commitment/iavl/config.go Outdated Show resolved Hide resolved
store/v2/options.go Outdated Show resolved Hide resolved
store/v2/root/factory.go Outdated Show resolved Hide resolved
store/v2/go.mod Outdated Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 237ef66 and 7d6bdaa.

Files ignored due to path filters (1)
  • store/v2/go.sum is excluded by !**/*.sum
Files selected for processing (7)
  • runtime/v2/builder.go (4 hunks)
  • server/v2/store/commands.go (1 hunks)
  • server/v2/store/config.go (1 hunks)
  • store/v2/commitment/iavl/config.go (1 hunks)
  • store/v2/go.mod (2 hunks)
  • store/v2/options.go (1 hunks)
  • store/v2/root/factory.go (4 hunks)
Files skipped from review as they are similar to previous changes (6)
  • server/v2/store/commands.go
  • server/v2/store/config.go
  • store/v2/commitment/iavl/config.go
  • store/v2/go.mod
  • store/v2/options.go
  • store/v2/root/factory.go
Additional context used
Path-based instructions (1)
runtime/v2/builder.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

Additional comments not posted (8)
runtime/v2/builder.go (8)

10-10: Approved: Addition of viper import.

The addition of the viper package is appropriate for enhancing configuration management.


29-29: Approved: Addition of viper field to AppBuilder struct.

The addition of the viper field enhances the configuration management capabilities of the AppBuilder.


124-126: Approved: Integration of viper in the Build method.

The integration of viper for retrieving the home directory and other configurations is appropriate.


127-132: Approved: Reading store options using viper.

The logic for reading store options from the configuration using viper is correct.


134-137: Approved: Initialization of new database instance.

The initialization of a new database instance using db.NewDB is correct.


139-146: Approved: Creation of storeOptions variable.

The creation of the storeOptions variable encapsulates the necessary parameters for root store creation effectively.


147-149: Approved: Call to rootstore.CreateRootStore.

The call to rootstore.CreateRootStore with the storeOptions is correct.


Line range hint 151-192:
Approved: Initialization of appManagerBuilder.

The initialization of the appManagerBuilder with the necessary parameters is correct.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 7d6bdaa and 145dcc2.

Files ignored due to path filters (1)
  • runtime/v2/go.sum is excluded by !**/*.sum
Files selected for processing (5)
  • runtime/v2/builder.go (4 hunks)
  • runtime/v2/go.mod (5 hunks)
  • server/v2/go.mod (4 hunks)
  • simapp/v2/go.mod (1 hunks)
  • simapp/v2/simdv2/cmd/commands.go (2 hunks)
Files skipped from review as they are similar to previous changes (4)
  • runtime/v2/go.mod
  • server/v2/go.mod
  • simapp/v2/go.mod
  • simapp/v2/simdv2/cmd/commands.go
Additional context used
Path-based instructions (1)
runtime/v2/builder.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

Additional comments not posted (9)
runtime/v2/builder.go (9)

10-10: Import statement for Viper added

The import statement for github.com/spf13/viper is necessary for the configuration management added in this file.


29-29: New field viper added to AppBuilder struct

The new field viper of type *viper.Viper is added to the AppBuilder struct for configuration management.


127-129: Retrieve home directory from Viper configuration

The home directory is retrieved from the Viper configuration using v.GetString(FlagHome). Ensure that FlagHome is correctly set in the configuration.


130-135: Store options initialization

The store options are initialized from the Viper configuration. Ensure that the store.options section exists in the configuration file and is correctly formatted.


137-141: Database initialization

The database is initialized using db.NewDB. Ensure that the store.app-db-backend configuration is correctly set and supports the specified database type.


142-149: Store options encapsulation

The store options are encapsulated within the rootstore.FactoryOptions struct. This enhances the clarity and structure of the code.


150-153: Create root store

The CreateRootStore method is called with the encapsulated store options. Ensure that the method handles errors appropriately.


Line range hint 155-180:
App manager builder initialization

The appManagerBuilder is initialized with the necessary parameters for building the app manager. Ensure that the initialization is correct and handles errors appropriately.


Line range hint 182-186:
Build app manager

The Build method is called for appManager and the result is assigned to a.app.AppManager. Ensure that the method handles errors appropriately.

Copy link
Member

@julienrbrt julienrbrt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK!

server/v2/store/commands.go Outdated Show resolved Hide resolved
store/v2/root/factory.go Outdated Show resolved Hide resolved
runtime/v2/builder.go Outdated Show resolved Hide resolved
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 145dcc2 and 32249d3.

Files selected for processing (6)
  • runtime/v2/builder.go (4 hunks)
  • server/v2/store/commands.go (1 hunks)
  • server/v2/store/config.go (1 hunks)
  • store/v2/commitment/iavl/config.go (1 hunks)
  • store/v2/options.go (1 hunks)
  • store/v2/root/factory.go (4 hunks)
Files skipped from review as they are similar to previous changes (4)
  • server/v2/store/commands.go
  • server/v2/store/config.go
  • store/v2/commitment/iavl/config.go
  • store/v2/options.go
Additional context used
Path-based instructions (2)
store/v2/root/factory.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

runtime/v2/builder.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

Additional comments not posted (1)
store/v2/root/factory.go (1)

35-42: LGTM!

The Options struct is well-defined with appropriate tags and comments.

Comment on lines +52 to +68
func DefaultStoreOptions() Options {
return Options{
SSType: 0,
SCType: 0,
SCPruningOption: &store.PruningOption{
KeepRecent: 2,
Interval: 1,
},
SSPruningOption: &store.PruningOption{
KeepRecent: 2,
Interval: 1,
},
IavlConfig: &iavl.Config{
CacheSize: 100_000,
SkipFastStorageUpgrade: true,
},
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving the ensureDir function outside.

To enhance readability, consider moving the ensureDir function outside the CreateRootStore function.

func ensureDir(dir string) error {
	if err := os.MkdirAll(dir, 0o0755); err != nil {
		return fmt.Errorf("failed to create directory %s: %w", dir, err)
	}
	return nil
}

func CreateRootStore(opts *FactoryOptions) (store.RootStore, error) {
	var (
		ssDb      storage.Database
		ss        *storage.StorageStore
		sc        *commitment.CommitStore
		err       error
	)

	storeOpts := opts.Options
	// ... rest of the code
}

Comment on lines +127 to +148
v := a.viper
home := v.GetString(FlagHome)

storeOpts := rootstore.DefaultStoreOptions()
if err := v.Sub("store.options").Unmarshal(&storeOpts); err != nil {
return nil, fmt.Errorf("failed to store options: %w", err)
}

scRawDb, err := db.NewDB(db.DBType(v.GetString("store.app-db-backend")), "application", filepath.Join(home, "data"), nil)
if err != nil {
panic(err)
}

storeOptions := &rootstore.FactoryOptions{
Logger: a.app.logger,
RootDir: home,
Options: storeOpts,
StoreKeys: append(a.app.storeKeys, "stf"),
SCRawDB: scRawDb,
}
a.storeOptions = storeOptions

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improve error handling for v.Sub("store.options").Unmarshal(&storeOpts).

To provide more context, improve the error handling for v.Sub("store.options").Unmarshal(&storeOpts).

-	if err := v.Sub("store.options").Unmarshal(&storeOpts); err != nil {
-		return nil, fmt.Errorf("failed to store options: %w", err)
+	sub := v.Sub("store.options")
+	if sub == nil {
+		return nil, fmt.Errorf("failed to get sub configuration for store.options")
+	}
+	if err := sub.Unmarshal(&storeOpts); err != nil {
+		return nil, fmt.Errorf("failed to unmarshal store options: %w", err)
	}
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
v := a.viper
home := v.GetString(FlagHome)
storeOpts := rootstore.DefaultStoreOptions()
if err := v.Sub("store.options").Unmarshal(&storeOpts); err != nil {
return nil, fmt.Errorf("failed to store options: %w", err)
}
scRawDb, err := db.NewDB(db.DBType(v.GetString("store.app-db-backend")), "application", filepath.Join(home, "data"), nil)
if err != nil {
panic(err)
}
storeOptions := &rootstore.FactoryOptions{
Logger: a.app.logger,
RootDir: home,
Options: storeOpts,
StoreKeys: append(a.app.storeKeys, "stf"),
SCRawDB: scRawDb,
}
a.storeOptions = storeOptions
v := a.viper
home := v.GetString(FlagHome)
storeOpts := rootstore.DefaultStoreOptions()
sub := v.Sub("store.options")
if sub == nil {
return nil, fmt.Errorf("failed to get sub configuration for store.options")
}
if err := sub.Unmarshal(&storeOpts); err != nil {
return nil, fmt.Errorf("failed to unmarshal store options: %w", err)
}
scRawDb, err := db.NewDB(db.DBType(v.GetString("store.app-db-backend")), "application", filepath.Join(home, "data"), nil)
if err != nil {
panic(err)
}
storeOptions := &rootstore.FactoryOptions{
Logger: a.app.logger,
RootDir: home,
Options: storeOpts,
StoreKeys: append(a.app.storeKeys, "stf"),
SCRawDB: scRawDb,
}
a.storeOptions = storeOptions

@hieuvubk hieuvubk added this pull request to the merge queue Jul 23, 2024
Merged via the queue into main with commit 0ca95ee Jul 23, 2024
70 of 74 checks passed
@hieuvubk hieuvubk deleted the hieu/prune-v2 branch July 23, 2024 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

simapp/v2 needs an implementation of CLI pruning commands
6 participants