Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(server/v2) Add prune cmd to serverv2 #20736

Merged
merged 80 commits into from
Jul 23, 2024
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
8f89f81
add prune func
hieuvubk Jun 20, 2024
462dd13
add prune cmd
hieuvubk Jun 20, 2024
33f674a
add to cmd set
hieuvubk Jun 20, 2024
0c3c6ce
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 20, 2024
45be505
lint
hieuvubk Jun 20, 2024
4769135
move prune to store component
hieuvubk Jun 20, 2024
d4927b4
rename
hieuvubk Jun 20, 2024
b99e554
add to simapp
hieuvubk Jun 20, 2024
44d609c
lint
hieuvubk Jun 20, 2024
8c4822e
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 20, 2024
63b1b03
move flags to component
hieuvubk Jun 20, 2024
0907699
Merge branch 'hieu/prune-v2' of https://github.com/cosmos/cosmos-sdk …
hieuvubk Jun 20, 2024
ac93a66
New func
hieuvubk Jun 20, 2024
40eab2f
unexpected change
hieuvubk Jun 20, 2024
974e3f9
lint
hieuvubk Jun 20, 2024
3ae993a
write store config to app.toml
hieuvubk Jun 20, 2024
7b5fda6
read store config in Init
hieuvubk Jun 20, 2024
8071c9f
update logic: read pruning options from flag or app.toml
hieuvubk Jun 20, 2024
b6a6cfe
lint
hieuvubk Jun 20, 2024
e70f3db
go mod
hieuvubk Jun 20, 2024
c6f36fe
make as pointer
hieuvubk Jun 20, 2024
19cf97d
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 20, 2024
36db1e7
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 21, 2024
62ab829
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 21, 2024
323250a
Init only return err
hieuvubk Jun 21, 2024
bc06fdf
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 24, 2024
dab94a4
merge & resolve conflict
hieuvubk Jun 24, 2024
e0bc1d7
remove v2/store go.mod
hieuvubk Jun 24, 2024
feb346d
simapp go mod tidy
hieuvubk Jun 24, 2024
ace696d
upstream prune v1 options helper
hieuvubk Jun 24, 2024
1bd995d
address comment
hieuvubk Jun 24, 2024
5099084
go mod tidy
hieuvubk Jun 24, 2024
a70891c
lint
hieuvubk Jun 24, 2024
355e8e2
remove store v1
hieuvubk Jun 24, 2024
916627d
Merge branch 'main' into hieu/prune-v2
hieuvubk Jun 25, 2024
46334c8
unused
hieuvubk Jun 25, 2024
0c28588
move AppI, AppCreator, CLIConfig to core/server
hieuvubk Jun 26, 2024
eda5037
move to store/v2
hieuvubk Jun 26, 2024
8becfe3
update simapp
hieuvubk Jun 26, 2024
ccc0976
lint
hieuvubk Jun 26, 2024
75906b0
merge main
hieuvubk Jun 26, 2024
0676698
go mod tidy all
hieuvubk Jun 26, 2024
1c08a6d
collections go mod
hieuvubk Jun 26, 2024
2d3e581
viper as AppOptions
hieuvubk Jun 27, 2024
07a53d1
abtract HasCLICommands
hieuvubk Jun 27, 2024
31c3da4
go mod tidy all
hieuvubk Jun 27, 2024
93e797b
merge & resolve conflict
hieuvubk Jun 27, 2024
7ca4458
lint
hieuvubk Jun 27, 2024
316a9d2
resolve conflict
hieuvubk Jun 27, 2024
f1d40fc
update x/tx go mod
hieuvubk Jun 27, 2024
57682b5
merge & resolve conflict
hieuvubk Jun 28, 2024
ee6057b
merge main & resolve conflict
hieuvubk Jul 8, 2024
4e871f1
create root store without app instance
hieuvubk Jul 8, 2024
ad013c5
update
hieuvubk Jul 8, 2024
1984592
go mod tidy all
hieuvubk Jul 8, 2024
d81532a
lint
hieuvubk Jul 8, 2024
3dba15d
print err format
hieuvubk Jul 8, 2024
ccc3722
reduce go mod change
hieuvubk Jul 8, 2024
6b5b25a
bring back to server/v2
hieuvubk Jul 16, 2024
9a992b7
lint
hieuvubk Jul 16, 2024
a7a38ed
Factory options take viper as store options
hieuvubk Jul 17, 2024
786995a
should work
hieuvubk Jul 17, 2024
58fface
resolve conflict
hieuvubk Jul 17, 2024
43b23c1
handle flags & print
hieuvubk Jul 18, 2024
158f3e5
remove FactoryOptions from AppInputs & Build from viper
hieuvubk Jul 18, 2024
3fa1187
refactor
hieuvubk Jul 18, 2024
5dcd3c4
handle keep-recent flag
hieuvubk Jul 18, 2024
c602198
home flag
hieuvubk Jul 18, 2024
f833ca4
runtime AppInputs take a viper
hieuvubk Jul 19, 2024
a2cd556
resolve conflict
hieuvubk Jul 19, 2024
1a56ec6
refactor
hieuvubk Jul 19, 2024
67cf8b4
lint
hieuvubk Jul 19, 2024
237ef66
godoc
hieuvubk Jul 19, 2024
ded5c63
remove viper dep
hieuvubk Jul 22, 2024
30e521f
multiple db back end
hieuvubk Jul 23, 2024
700d59c
lint
hieuvubk Jul 23, 2024
7d6bdaa
add comment
hieuvubk Jul 23, 2024
145dcc2
Merge branch 'main' into hieu/prune-v2
hieuvubk Jul 23, 2024
3d01423
comments
hieuvubk Jul 23, 2024
32249d3
DefaultStoreOptions to store/v2/root
hieuvubk Jul 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion server/v2/cometbft/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ func (s *CometBFTServer[T]) StartCmdFlags() pflag.FlagSet {
return flags
}

func (s *CometBFTServer[T]) CLICommands() serverv2.CLIConfig {
func (s *CometBFTServer[T]) CLICommands(_ serverv2.AppCreator[T]) serverv2.CLIConfig {
return serverv2.CLIConfig{
Commands: []*cobra.Command{
s.StatusCommand(),
Expand Down
2 changes: 1 addition & 1 deletion server/v2/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ func Commands(rootCmd *cobra.Command, newApp AppCreator[transaction.Tx], logger
},
}

cmds := server.CLICommands()
cmds := server.CLICommands(newApp)
cmds.Commands = append(cmds.Commands, startCmd)

return cmds, nil
Expand Down
10 changes: 5 additions & 5 deletions server/v2/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ type ServerComponent[T transaction.Tx] interface {

// HasCLICommands is a server module that has CLI commands.
type HasCLICommands interface {
CLICommands() CLIConfig
CLICommands(AppCreator[transaction.Tx]) CLIConfig
}

// HasConfig is a server module that has a config.
Expand Down Expand Up @@ -118,13 +118,13 @@ func (s *Server) Stop(ctx context.Context) error {
}

// CLICommands returns all CLI commands of all components.
func (s *Server) CLICommands() CLIConfig {
func (s *Server) CLICommands(appCreator AppCreator[transaction.Tx]) CLIConfig {
commands := CLIConfig{}
for _, mod := range s.components {
if climod, ok := mod.(HasCLICommands); ok {
commands.Commands = append(commands.Commands, climod.CLICommands().Commands...)
commands.Queries = append(commands.Queries, climod.CLICommands().Queries...)
commands.Txs = append(commands.Txs, climod.CLICommands().Txs...)
commands.Commands = append(commands.Commands, climod.CLICommands(appCreator).Commands...)
commands.Queries = append(commands.Queries, climod.CLICommands(appCreator).Queries...)
commands.Txs = append(commands.Txs, climod.CLICommands(appCreator).Txs...)
hieuvubk marked this conversation as resolved.
Show resolved Hide resolved
}
}

Expand Down
125 changes: 125 additions & 0 deletions server/v2/store/commands.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
package store
tac0turtle marked this conversation as resolved.
Show resolved Hide resolved

import (
"fmt"
"strings"

"github.com/spf13/cast"
"github.com/spf13/cobra"
"github.com/spf13/viper"

"cosmossdk.io/core/transaction"
"cosmossdk.io/log"
serverv2 "cosmossdk.io/server/v2"
pruningtypes "cosmossdk.io/store/pruning/types"
storev2 "cosmossdk.io/store/v2"

"github.com/cosmos/cosmos-sdk/version"
hieuvubk marked this conversation as resolved.
Show resolved Hide resolved
)

// QueryBlockResultsCmd implements the default command for a BlockResults query.
func (s StoreComponent) PrunesCmd(appCreator serverv2.AppCreator[transaction.Tx]) *cobra.Command {
cmd := &cobra.Command{
Use: "prune [pruning-method]",
Short: "Prune app history states by keeping the recent heights and deleting old heights",
Long: `Prune app history states by keeping the recent heights and deleting old heights.
The pruning option is provided via the 'pruning' argument or alternatively with '--pruning-keep-recent'

- default: the last 362880 states are kept
- nothing: all historic states will be saved, nothing will be deleted (i.e. archiving node)
- everything: 2 latest states will be kept
- custom: allow pruning options to be manually specified through 'pruning-keep-recent'

Note: When the --app-db-backend flag is not specified, the default backend type is 'goleveldb'.
Supported app-db-backend types include 'goleveldb', 'rocksdb', 'pebbledb'.`,
Example: fmt.Sprintf("%s prune custom --pruning-keep-recent 100 --app-db-backend 'goleveldb'", version.AppName),
Args: cobra.RangeArgs(0, 1),
RunE: func(cmd *cobra.Command, args []string) error {
// bind flags to the Context's Viper so we can get pruning options.
vp := viper.New()
if err := vp.BindPFlags(cmd.Flags()); err != nil {
return err
}
if err := vp.BindPFlags(cmd.PersistentFlags()); err != nil {
return err
}

// use the first argument if present to set the pruning method
if len(args) > 0 {
vp.Set(FlagPruning, args[0])
} else {
vp.Set(FlagPruning, pruningtypes.PruningOptionDefault)
}
pruningOptions, err := getPruningOptionsFromFlags(vp)
if err != nil {
return err
}

cmd.Printf("get pruning options from command flags, strategy: %v, keep-recent: %v\n",
pruningOptions.Strategy,
pruningOptions.KeepRecent,
)

logger := log.NewLogger(cmd.OutOrStdout())
app := appCreator(logger, vp)
store := app.GetStore()

rootStore, ok := store.(storev2.RootStore)
if !ok {
return fmt.Errorf("currently only support the pruning of rootmulti.Store type")
}
latestHeight, err := rootStore.GetLatestVersion()
if err != nil {
return err
}

// valid heights should be greater than 0.
if latestHeight <= 0 {
return fmt.Errorf("the database has no valid heights to prune, the latest height: %v", latestHeight)
}

pruningHeight := latestHeight - pruningOptions.KeepRecent
cmd.Printf("pruning heights up to %v\n", pruningHeight)

err = rootStore.Prune(pruningHeight)
if err != nil {
return err
}

cmd.Println("successfully pruned the application root multi stores")
return nil
},
}

cmd.Flags().String(FlagAppDBBackend, "", "The type of database for application and snapshots databases")
cmd.Flags().Uint64(FlagPruningKeepRecent, 0, "Number of recent heights to keep on disk (ignored if pruning is not 'custom')")
cmd.Flags().Uint64(FlagPruningInterval, 10,
`Height interval at which pruned heights are removed from disk (ignored if pruning is not 'custom'),
this is not used by this command but kept for compatibility with the complete pruning options`)

return cmd
}

func getPruningOptionsFromFlags(v *viper.Viper) (pruningtypes.PruningOptions, error) {
strategy := strings.ToLower(cast.ToString(v.Get(FlagPruning)))

switch strategy {
case pruningtypes.PruningOptionDefault, pruningtypes.PruningOptionNothing, pruningtypes.PruningOptionEverything:
return pruningtypes.NewPruningOptionsFromString(strategy), nil

case pruningtypes.PruningOptionCustom:
opts := pruningtypes.NewCustomPruningOptions(
cast.ToUint64(v.Get(FlagPruningKeepRecent)),
cast.ToUint64(v.Get(FlagPruningInterval)),
)

if err := opts.Validate(); err != nil {
return opts, fmt.Errorf("invalid custom pruning options: %w", err)
}

return opts, nil

default:
return pruningtypes.PruningOptions{}, fmt.Errorf("unknown pruning strategy %s", strategy)
}
}
8 changes: 8 additions & 0 deletions server/v2/store/flags.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package store

const (
tac0turtle marked this conversation as resolved.
Show resolved Hide resolved
FlagPruning = "pruning"
FlagAppDBBackend = "app-db-backend"
FlagPruningKeepRecent = "pruning-keep-recent"
FlagPruningInterval = "pruning-interval"
)
72 changes: 72 additions & 0 deletions server/v2/store/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
module cosmossdk.io/server/v2/store
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the new go.mod? can we use store/v2 as the home for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now this new server module is just for saving prune cmd & maybe snapshot in the future.
@julienrbrt any idea?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also should we move it back to cometbft since we already have some snapshot helper there

Copy link
Member

@julienrbrt julienrbrt Jun 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We were thinking about putting it in the cometbft server first indeed, but pruning isn't cometbft specific, so we need a better place for it.

We may not want to add a server/v2 and cobra dep in store/v2. An additional server did seem a good idea.

However, given that api and grpc haven't their own go.mod, maybe let's just keep the separate server component but kill the go.mod 👍🏾


go 1.22.2

replace (
cosmossdk.io/api => ../../../api
cosmossdk.io/core => ../../../core
cosmossdk.io/core/testing => ../../../core/testing
cosmossdk.io/depinject => ../../../depinject
cosmossdk.io/log => ../../../log
cosmossdk.io/server/v2 => ../
cosmossdk.io/server/v2/appmanager => ../appmanager
cosmossdk.io/store => ../../../store
cosmossdk.io/store/v2 => ../../../store/v2
cosmossdk.io/x/accounts => ../../../x/accounts
cosmossdk.io/x/auth => ../../../x/auth
cosmossdk.io/x/bank => ../../../x/bank
cosmossdk.io/x/consensus => ../../../x/consensus
cosmossdk.io/x/staking => ../../../x/staking
github.com/cosmos/cosmos-sdk => ../../../
)

require (
cosmossdk.io/core v0.12.1-0.20231114100755-569e3ff6a0d7
cosmossdk.io/log v1.3.1
cosmossdk.io/server/v2 v2.0.0-00010101000000-000000000000
cosmossdk.io/store v1.1.1-0.20240418092142-896cdf1971bc
cosmossdk.io/store/v2 v2.0.0-00010101000000-000000000000
github.com/cosmos/cosmos-sdk v0.51.0
github.com/spf13/cast v1.6.0
github.com/spf13/cobra v1.8.1
github.com/spf13/viper v1.19.0
)

require (
cosmossdk.io/errors v1.0.1 // indirect
cosmossdk.io/server/v2/appmanager v0.0.0-00010101000000-000000000000 // indirect
github.com/cosmos/gogoproto v1.5.0 // indirect
github.com/cosmos/ics23/go v0.10.0 // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/go-metrics v0.5.3 // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/rs/zerolog v1.33.0 // indirect
github.com/sagikazarmark/locafero v0.4.0 // indirect
github.com/sagikazarmark/slog-shim v0.1.0 // indirect
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.24.0 // indirect
golang.org/x/exp v0.0.0-20240531132922-fd00a4e0eefc // indirect
golang.org/x/sync v0.7.0 // indirect
golang.org/x/sys v0.21.0 // indirect
golang.org/x/text v0.16.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 // indirect
google.golang.org/grpc v1.64.0 // indirect
google.golang.org/protobuf v1.34.2 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)
Loading
Loading