Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pebble as valid value for --db-type. #2244

Merged
merged 85 commits into from
Oct 31, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
d24ee4c
initial commit
Sep 11, 2023
d578d00
Merge remote-tracking branch 'upstream/dev' into pebble
Sep 11, 2023
db2eeeb
update pebble dep
Sep 11, 2023
0983042
Merge remote-tracking branch 'upstream/dev' into pebble
Sep 12, 2023
528cc2d
appease linter
Sep 12, 2023
1bcfa65
nit
Sep 12, 2023
73a934f
nit
Sep 12, 2023
1e094b4
nit
Sep 12, 2023
5f39300
Merge branch 'dev' into pebble
Sep 12, 2023
d937e6c
Merge branch 'dev' into pebble
Sep 15, 2023
721275e
nit
Sep 15, 2023
5a8d855
Merge branch 'dev' into pebble
Sep 18, 2023
9f7665b
fix tests
Sep 18, 2023
1db44c4
nit
Sep 18, 2023
7e4cf61
Merge branch 'dev' into pebble
Sep 19, 2023
4051e88
nit
Sep 19, 2023
4bc0630
Merge branch 'dev' into pebble
Sep 22, 2023
662975f
Merge branch 'dev' into pebble
Sep 25, 2023
cd0a8e2
appease linter
Sep 25, 2023
261e710
Merge branch 'dev' into pebble
dhrubabasu Sep 27, 2023
59a8615
Merge remote-tracking branch 'upstream/dev' into pebble
Oct 2, 2023
a710f54
remove TODO
Oct 2, 2023
f21c81d
remove unneeded var
Oct 2, 2023
cc7a15e
unexport error
Oct 2, 2023
f9192ed
atomic bool --> bool
Oct 2, 2023
155edc9
merged
StephenButtolph Oct 24, 2023
72a249d
match pebble version
StephenButtolph Oct 24, 2023
4b08b25
fix test:
Oct 25, 2023
b8fa698
change pebble opening log; edit comment
Oct 25, 2023
a1b7fba
pebbledb.New nit
Oct 25, 2023
6b93c90
Lock --> RLock in Put, Delete
Oct 25, 2023
610e7f8
iterator nits
Oct 25, 2023
fc91cd1
Compact Lock --> RLock
Oct 25, 2023
5aff591
prefixBounds --> keyRange and prefixToUpperBound
Oct 25, 2023
49499fa
Merge remote-tracking branch 'upstream/dev' into pebble
Oct 25, 2023
bba3048
remove unneeded iterator creator code
Oct 25, 2023
3ffc6a8
remove unneeded check in iter.Next
Oct 25, 2023
a8ff7b8
remove unused var
Oct 25, 2023
c6bacc7
change switch order in iter.Next to not overwrite iter error
Oct 25, 2023
b886790
remove unneeded lock grab and db closed check in iter.Next
Oct 25, 2023
f771665
prevent Compact panic
Oct 25, 2023
3412216
fix default MaxConcurrentCompactions; fix compaction iterator leak; a…
Oct 25, 2023
f0e3f2e
cleanup and comments
Oct 25, 2023
c4d1a87
add Key and Value invariant on database.Iterator
Oct 25, 2023
d379e77
spelling is hard
Oct 25, 2023
8eea9eb
make pebble.New take in config bytes instead of config
Oct 25, 2023
c350116
Merge remote-tracking branch 'upstream/dev' into pebble
Oct 25, 2023
b64de08
os.MkDirTemp --> t.TempDir
Oct 25, 2023
b1768c1
remove unneeded file removal
Oct 25, 2023
afd88e3
Close db
StephenButtolph Oct 25, 2023
b360819
add flags for pebble; only unmarshal config bytes if given
Oct 25, 2023
6f89805
Merge remote-tracking branch 'upstream/dev' into pebble-2
Oct 25, 2023
9b4343f
Merge branch 'dev' into pebble-2
Oct 26, 2023
417785f
comments
Oct 26, 2023
da99730
Merge branch 'pebble-2' of github.com:ava-labs/avalanchego into pebble-2
Oct 26, 2023
fbfdc6d
Merge branch 'dev' into pebble-2
Oct 26, 2023
70d3420
Merge branch 'dev' into pebble-2
Oct 27, 2023
f41856a
Merge branch 'dev' into pebble-2
Oct 27, 2023
b49340b
nit move name declaration
Oct 30, 2023
c18a208
Merge branch 'dev' into pebble-2
Oct 30, 2023
3dca091
remove db manager
Oct 30, 2023
a131092
fix initDatabase
Oct 30, 2023
ce33816
fix initDatabase
Oct 30, 2023
0b66354
revert go.mod and go.sum changes
Oct 30, 2023
f729156
remove DBServer proto type
Oct 30, 2023
d0ff782
revert scripts changes
Oct 30, 2023
4ea6944
revert byte change
Oct 30, 2023
3e6841f
nit
Oct 30, 2023
af1bc23
change healthy reporting
Oct 30, 2023
0891312
Merge branch 'dev' into remove-db-manager
Oct 30, 2023
e6b970a
update coreth dep
Oct 31, 2023
8ee8821
Merge branch 'remove-db-manager' into pebble-no-db-manager
Oct 31, 2023
ba9fac7
update level/pebble creation
Oct 31, 2023
a0b4ef8
remove useless vm.connCloser.Close call
Oct 31, 2023
4fd170d
fix camelCase
Oct 31, 2023
81bb71e
nits
Oct 31, 2023
3987bf5
fix database prefixing
Oct 31, 2023
a8f1fe1
Merge remote-tracking branch 'upstream/dev' into remove-db-manager
Oct 31, 2023
3a71f38
Merge branch 'dev' into pebble-2
Oct 31, 2023
63efa76
make prefix var
Oct 31, 2023
887c094
Merge branch 'remove-db-manager' into pebble-no-db-manager
Oct 31, 2023
10a9a56
Merge remote-tracking branch 'upstream/pebble-2' into pebble-no-db-ma…
Oct 31, 2023
8feabe9
Reduce diff + complexity
StephenButtolph Oct 31, 2023
de99f6e
include comment
StephenButtolph Oct 31, 2023
f562920
merged
StephenButtolph Oct 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion config/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (

"github.com/ava-labs/avalanchego/database/leveldb"
"github.com/ava-labs/avalanchego/database/memdb"
"github.com/ava-labs/avalanchego/database/pebble"
"github.com/ava-labs/avalanchego/genesis"
"github.com/ava-labs/avalanchego/snow/consensus/snowball"
"github.com/ava-labs/avalanchego/trace"
Expand Down Expand Up @@ -103,7 +104,7 @@ func addNodeFlags(fs *pflag.FlagSet) {
fs.Uint64(AddSubnetDelegatorFeeKey, genesis.LocalParams.AddSubnetDelegatorFee, "Transaction fee, in nAVAX, for transactions that add new subnet delegators")

// Database
fs.String(DBTypeKey, leveldb.Name, fmt.Sprintf("Database type to use. Should be one of {%s, %s}", leveldb.Name, memdb.Name))
fs.String(DBTypeKey, leveldb.Name, fmt.Sprintf("Database type to use. Must be one of {%s, %s, %s}", leveldb.Name, memdb.Name, pebble.Name))
fs.String(DBPathKey, defaultDBDir, "Path to database directory")
fs.String(DBConfigFileKey, "", fmt.Sprintf("Path to database config file. Ignored if %s is specified", DBConfigContentKey))
fs.String(DBConfigContentKey, "", "Specifies base64 encoded database config content")
Expand Down
20 changes: 13 additions & 7 deletions database/pebble/db.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,13 @@ import (
"github.com/ava-labs/avalanchego/utils/units"
)

// pebbleByteOverHead is the number of bytes of constant overhead that
// should be added to a batch size per operation.
const pebbleByteOverHead = 8
const (
Name = "pebble"

// pebbleByteOverHead is the number of bytes of constant overhead that
// should be added to a batch size per operation.
pebbleByteOverHead = 8
)

var (
_ database.Database = (*Database)(nil)
Expand Down Expand Up @@ -73,10 +77,12 @@ type Config struct {
}

// TODO: Add metrics
func New(file string, configBytes []byte, log logging.Logger, _ string, _ prometheus.Registerer) (*Database, error) {
var cfg Config
if err := json.Unmarshal(configBytes, &cfg); err != nil {
return nil, err
func New(file string, configBytes []byte, log logging.Logger, _ string, _ prometheus.Registerer) (database.Database, error) {
cfg := DefaultConfig
if len(configBytes) > 0 {
if err := json.Unmarshal(configBytes, &cfg); err != nil {
return nil, err
}
}

opts := &pebble.Options{
Expand Down
2 changes: 1 addition & 1 deletion database/pebble/db_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ func newDB(t testing.TB) *Database {
folder := t.TempDir()
db, err := New(folder, DefaultConfigBytes, logging.NoLog{}, "pebble", prometheus.NewRegistry())
require.NoError(t, err)
return db
return db.(*Database)
}

func TestInterface(t *testing.T) {
Expand Down
15 changes: 13 additions & 2 deletions node/node.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ import (
"github.com/ava-labs/avalanchego/database/leveldb"
"github.com/ava-labs/avalanchego/database/memdb"
"github.com/ava-labs/avalanchego/database/meterdb"
"github.com/ava-labs/avalanchego/database/pebble"
"github.com/ava-labs/avalanchego/database/prefixdb"
"github.com/ava-labs/avalanchego/genesis"
"github.com/ava-labs/avalanchego/ids"
Expand Down Expand Up @@ -503,20 +504,30 @@ func (n *Node) initDatabase() error {
// start the db
switch n.Config.DatabaseConfig.Name {
case leveldb.Name:
// Prior to v1.10.15, the only on-disk database was leveldb, and its
// files went to [dbPath]/[networkID]/v1.4.5.
dbPath := filepath.Join(n.Config.DatabaseConfig.Path, version.CurrentDatabase.String())
var err error
n.DB, err = leveldb.New(dbPath, n.Config.DatabaseConfig.Config, n.Log, "db_internal", n.MetricsRegisterer)
if err != nil {
return fmt.Errorf("couldn't create db at %s: %w", dbPath, err)
return fmt.Errorf("couldn't create leveldb at %s: %w", dbPath, err)
}
case memdb.Name:
n.DB = memdb.New()
case pebble.Name:
dbPath := filepath.Join(n.Config.DatabaseConfig.Path, pebble.Name)
var err error
n.DB, err = pebble.New(dbPath, n.Config.DatabaseConfig.Config, n.Log, "db_internal", n.MetricsRegisterer)
if err != nil {
return fmt.Errorf("couldn't create pebbledb at %s: %w", dbPath, err)
}
default:
return fmt.Errorf(
"db-type was %q but should have been one of {%s, %s}",
"db-type was %q but should have been one of {%s, %s, %s}",
n.Config.DatabaseConfig.Name,
leveldb.Name,
memdb.Name,
pebble.Name,
)
}

Expand Down
Loading