Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks for gas pricing #634

Merged
merged 5 commits into from
Oct 8, 2021
Merged

Benchmarks for gas pricing #634

merged 5 commits into from
Oct 8, 2021

Conversation

ethanfrey
Copy link
Member

@ethanfrey ethanfrey commented Oct 7, 2021

Prepares #566
Prepares #631

  • Run Instantiation (pinned/unpinned) benchmarks in CI
  • Run secp256k1 verification benchmarks in CI (as reference for the 1 TerraGas/ms wasm definition)
  • Run compilation benchmarks in CI
  • Calculate new gas prices

I will update prices in a new PR, once we agree on the benchmarks.

@ethanfrey
Copy link
Member Author

ethanfrey commented Oct 7, 2021

Doing calcs here first...
Based on numbers from https://app.circleci.com/pipelines/github/CosmWasm/wasmd/1470/workflows/6389e6b6-0092-4e56-a592-8a2eed742771/jobs/4517

Set GasMultiplier to 280_000_000 (Wasm Gas per SDK Gas)

Secp timing: 281877 ns/call.
1.000.000 ns/ms * 1000gas/call / 281877 ns/call = 3547.64 sdk gas/ms
1.000.000.000.000 wasm gas /ms / 3547.64 sdk gas/ms = 281.877.000 wasm gas / sdk gas 

Set Instantiate Cost to 30_000:

Unpinned: 8614986 ns/op
Pinned: 171230 ns/op
Overhead = 8614986 - 171230 = 8443756 ns

Normalize with secp256k1:

8443756 ns/op * 1000gas /  281877 ns = 29955 gas/op

Compiling Cost to 1.2 (round to 1 or 2?)

42381402 ns / 124838 bytes = 339.5 ns/bytes
93803579 ns / 274438 bytes = 341.8 ns/byte
68310515 ns / 208793 bytes = 327.2 ns/byte
Average: 335 ns/byte

335 ns/byte * 1000 gas / 281877ns = 1.2 gas/byte

@ethanfrey
Copy link
Member Author

Interesting that we were actually overcharging previously. Mildly for Instantiate/Compile but almost 20x for Execution.

Old:

	// Please note that all gas prices returned to wasmvm should have this multiplied.
	DefaultGasMultiplier uint64 = 15_000_000
	// DefaultInstanceCost is how much SDK gas we charge each time we load a WASM instance.
	// Creating a new instance is costly, and this helps put a recursion limit to contracts calling contracts.
	DefaultInstanceCost uint64 = 40_000
	// DefaultCompileCost is how much SDK gas is charged *per byte* for compiling WASM code.
	DefaultCompileCost uint64 = 2

New (leaving compile cost at 2 as it does take some disk space also):

	// Please note that all gas prices returned to wasmvm should have this multiplied.
	DefaultGasMultiplier uint64 = 280_000_000
	// DefaultInstanceCost is how much SDK gas we charge each time we load a WASM instance.
	// Creating a new instance is costly, and this helps put a recursion limit to contracts calling contracts.
	DefaultInstanceCost uint64 = 30_000
	// DefaultCompileCost is how much SDK gas is charged *per byte* for compiling WASM code.
	DefaultCompileCost uint64 = 2

@ethanfrey ethanfrey marked this pull request as ready for review October 7, 2021 14:47
@ethanfrey ethanfrey requested a review from alpe as a code owner October 7, 2021 14:47
@ethanfrey
Copy link
Member Author

I had a discussion with @webmaster128 and he was concerned we were possibly comparing against an outlier and this may not be the same as measuring with storage gas. The problem with storage is it can vary a lot. Size of DB, "block"/batch size, size of nodes.

The only real benchmark I could find was in tendermint/iavl - most of it my code from 2017. I selected 1.000.000 entries with 16byte keys 40byte values and a batch size of 100 writes. As something to compare against.

BenchmarkLarge/goleveldb-1000000-100-16-40/update-12               10000            236901 ns/op           71404 B/op        587 allocs/op

236901ns for 2080 gas (2000 + 2*40) = 114ns/gas

On the same computer, the secp256k1 measurement gave:

BenchmarkGasNormalization-12                6206            191987 ns/op

or 192ns/gas.

Let's be nice and say the secp256k1 is charged 2x compared to storage operations. Thus, we can add such an adjustment factor in the pricing

@ethanfrey
Copy link
Member Author

This gives us:

Secp timing:

281877 ns/call / 1000 gas/call = 282 ns/gas
282ns/gas / 2 (adjustment for secp overcharge) = 141ns/gas

We can use that as the proper timing for an sdk gas.

That would give us:

1.000.000 wasm gas / ns * 141 ns / sdk gas = 141.000.000 sdk gas / wasm gas (GasMultiplier)

@ethanfrey
Copy link
Member Author

Carrying this on, it would mean InstantiateCost is 30.000*2 = 60.000

And CompileCostPerByte is 1.2 * 2 = 3 gas/byte

Leaving us:

	// Please note that all gas prices returned to wasmvm should have this multiplied.
	DefaultGasMultiplier uint64 = 140_000_000
	// DefaultInstanceCost is how much SDK gas we charge each time we load a WASM instance.
	// Creating a new instance is costly, and this helps put a recursion limit to contracts calling contracts.
	DefaultInstanceCost uint64 = 60_000
	// DefaultCompileCost is how much SDK gas is charged *per byte* for compiling WASM code.
	DefaultCompileCost uint64 = 3

Copy link
Contributor

@alpe alpe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work. added some minor nits

x/wasm/keeper/bench_test.go Outdated Show resolved Hide resolved
x/wasm/keeper/bench_test.go Outdated Show resolved Hide resolved
@alpe alpe mentioned this pull request Oct 8, 2021
@alpe alpe merged commit 08ee11d into master Oct 8, 2021
@alpe alpe deleted the 631-benchmark-for-gas-multiplier branch October 8, 2021 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants