reduce memory rusage during test runs

[holiman]

It would be neat to reduce the memory usage when running tests, making it run smoother on both CI and when running on your laptop.

Here's an allocation chart of running all tests in the les folder, I'll add some other parts later. I think there are a lot of low-hanging fruits to pick, e.g. like #21882 (comment).

[holiman]

core package

      flat  flat%   sum%        cum   cum%
    1.59GB 19.52% 19.52%     1.60GB 19.58%  github.com/syndtr/goleveldb/leveldb/memdb.New
    1.47GB 18.03% 37.56%     1.47GB 18.03%  github.com/steakknife/bloomfilter.newBits (inline)
    1.30GB 15.88% 53.44%     2.32GB 28.39%  github.com/VictoriaMetrics/fastcache.New
    1.02GB 12.50% 65.95%     1.02GB 12.50%  github.com/VictoriaMetrics/fastcache.(*bucket).Init
    0.75GB  9.22% 75.16%     0.75GB  9.22%  github.com/ethereum/go-ethereum/ethdb/memorydb.(*Database).NewIterato

1.30GB 15.88% 53.44% 2.32GB 28.39% github.com/VictoriaMetrics/fastcache.New -- this seems to be from the codeCache (64Mb) that gets created when creating a state database:

func NewDatabaseWithConfig(db ethdb.Database, config *trie.Config) Database {
	csc, _ := lru.New(codeSizeCacheSize)
	return &cachingDB{
		db:            trie.NewDatabaseWithConfig(db, config),
		codeSizeCache: csc,
		codeCache:     fastcache.New(codeCacheSize),

Might be that we don't even need such a cache for the tests, or if we do, maybe 16K is sufficient (EDIT: apparently 32Mb is the minimum cache capacity for fastcache)

[MariusVanDerWijden]

@holiman you seem to have a good setup running, could you test accounts/abi/bind for me? I think there's probably a lot to improve there

[holiman]

accounts/abi/bind package (pretty low on memory consumption already)

      flat  flat%   sum%        cum   cum%
 2069.73kB 11.41% 11.41%  2069.73kB 11.41%  bytes.makeSlice
    1634kB  9.01% 20.43%  2146.02kB 11.84%  github.com/VictoriaMetrics/fastcache.New
 1536.14kB  8.47% 28.90%  1536.14kB  8.47%  text/template/parse.(*Tree).newPipeline
 1056.33kB  5.83% 34.72%  1056.33kB  5.83%  regexp.(*bitState).reset
 1030.18kB  5.68% 40.40%  1030.18kB  5.68%  go/printer.(*printer).writeString
 1024.11kB  5.65% 46.05%  1024.11kB  5.65%  regexp/syntax.(*parser).newRegexp
 1024.06kB  5.65% 51.70%  1024.06kB  5.65%  text/template/parse.(*Tree).newText
  561.50kB  3.10% 54.80%   561.50kB  3.10%  golang.org/x/net/html.init
  516.01kB  2.85% 57.64%  2052.48kB 11.32%  go/printer.(*printer).nodeSize

[holiman]

Btw, here are the top 25 slowest tests on appveyor:

Test	Time
github.com/ethereum/go-ethereum/tests	213.115
github.com/ethereum/go-ethereum/eth/downloader	175.823
github.com/ethereum/go-ethereum/trie	128.769
github.com/ethereum/go-ethereum/p2p/discover	96.552
github.com/ethereum/go-ethereum/les	69.651
github.com/ethereum/go-ethereum/core	69.284
github.com/ethereum/go-ethereum/eth	51.893
github.com/ethereum/go-ethereum/accounts/keystore	21.879
github.com/ethereum/go-ethereum/node	16.591
github.com/ethereum/go-ethereum/internal/guide	16.445
github.com/ethereum/go-ethereum/consensus/ethash	14.975
github.com/ethereum/go-ethereum/core/bloombits	13.242
github.com/ethereum/go-ethereum/les/flowcontrol	12.15
github.com/ethereum/go-ethereum/core/state	11.868
github.com/ethereum/go-ethereum/eth/fetcher	11.732
github.com/ethereum/go-ethereum/rpc	10.654
github.com/ethereum/go-ethereum/cmd/geth	9.891
github.com/ethereum/go-ethereum/miner	9.577
github.com/ethereum/go-ethereum/p2p/discv5	8.888
github.com/ethereum/go-ethereum/metrics	6.25
github.com/ethereum/go-ethereum/crypto/bls12381	5.756
github.com/ethereum/go-ethereum/signer/core	4.472
github.com/ethereum/go-ethereum/core/state/snapshot	3.696
github.com/ethereum/go-ethereum/signer/fourbyte	3.651

[holiman]

This is pretty stale now, closing