Optimization package

[uprendis]

• add multidb framework, which allows to configure DBs and tables layout based on routing rules. It allows to tune performance by splitting/merging DBs, configuring DB type and DB cache amount
• add batching for genesis processing, which speeds up genesis blocks processing by ~2x
• DBs files descriptors limits is configurable and scaled by default according to OS process limit (fix File Descriptor limit #323)
• add caching for event IDs
• add caching for LLR voting weights
• during fullsync, LLR syncing is active only if events are already synced up
• fix nil cachescale issue after loading toml config file

[hadv]

we might rename this method to showDBStats() for generic name in multi db

[uprendis]

Yeah, but after we merge your changes for new statter. For now, this method indeed works only for leveldb

[hadv]

@uprendis I rebased the PR with develop branch and cannot start the opera node (from scratch)

panic: runtime error: invalid memory address or nil pointer dereference
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x4a44bb8]

goroutine 1 [running]:
github.com/Fantom-foundation/go-opera/gossip.(*handler).Stop(0xc00085e600)
        /Users/hadv/fantom/go-opera/gossip/handler.go:725 +0x1b8
github.com/Fantom-foundation/go-opera/gossip.(*Service).Stop(0xc0001fb000)
        /Users/hadv/fantom/go-opera/gossip/service.go:485 +0x139
github.com/ethereum/go-ethereum/node.(*Node).stopServices(0xc00013b450, {0xc0000c0fd0, 0x1, 0xc0006a2fc8?})
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:293 +0xbe
github.com/ethereum/go-ethereum/node.(*Node).Close(0xc00013b450)
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:213 +0x18e
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.makeNode.func3()
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:367 +0x38
panic({0x50fffa0, 0x6037d40})
        /usr/local/Cellar/go/1.18.3/libexec/src/runtime/panic.go:838 +0x207
github.com/Fantom-foundation/go-opera/gossip.(*Service).RecoverEVM(0xc0001fb000)
        /Users/hadv/fantom/go-opera/gossip/c_block_callbacks.go:493 +0xe4
github.com/Fantom-foundation/go-opera/gossip.(*Service).Start(0xc0001fb000)
        /Users/hadv/fantom/go-opera/gossip/service.go:436 +0x16b
github.com/ethereum/go-ethereum/node.(*Node).Start(0xc00013b450)
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/node/node.go:184 +0x3d7
github.com/ethereum/go-ethereum/cmd/utils.StartNode(0xc0000ba420, 0xc00013b450)
        /Users/hadv/go/pkg/mod/github.com/!fantom-foundation/[email protected]/cmd/utils/cmd.go:70 +0x2a
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.startNode(0xc0000ba420?, 0xc00013b450)
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:391 +0x54
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.lachesisMain(0xc0000ba420)
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:274 +0xe5
gopkg.in/urfave/cli%2ev1.HandleAction({0x5097ec0?, 0x53fbd98?}, 0xc000295c00?)
        /Users/hadv/go/pkg/mod/gopkg.in/urfave/[email protected]/app.go:490 +0x5a
gopkg.in/urfave/cli%2ev1.(*App).Run(0xc00021e9c0, {0xc00003c070, 0x7, 0x7})
        /Users/hadv/go/pkg/mod/gopkg.in/urfave/[email protected]/app.go:264 +0x71c
github.com/Fantom-foundation/go-opera/cmd/opera/launcher.Launch(...)
        /Users/hadv/fantom/go-opera/cmd/opera/launcher/launcher.go:252
main.main()
        /Users/hadv/fantom/go-opera/cmd/opera/main.go:11 +0x35

And a test FAILED aslo when running go test ./...

FAIL    github.com/Fantom-foundation/go-opera/cmd/opera/launcher [build failed]

[hadv]

And a test FAILED aslo when running go test ./...

FAIL    github.com/Fantom-foundation/go-opera/cmd/opera/launcher [build failed]

ah, this is just need to update the config_custom_test.go file after rebase code with develop branch

[uprendis]

@uprendis I rebased the PR with develop branch and cannot start the opera node (from scratch)

Got you, will rebase and fix it

[uprendis]

@uprendis I rebased the PR with develop branch and cannot start the opera node (from scratch)

It should be fixed now after 53eddab

[hadv]

It should be fixed now after 53eddab

Great, I worked for me!