kvserver: introduce Store.{logEngine,todoEngine}

[tbg]

Over the last few months, we have been chipping away¹ at separating the
engine interactions of the KV server in preparation for optionally
running with a separate log engine for the raft state.

Plenty of work remains to be able to safely split these engines apart,
or to even be able to do it at all, but now's a good time to take stock
and syntactically "boil the ocean" by forcing all users of
Store.engine to decide between three Engine fields - all for now
backed by the same Engine - now present on Store:

• LogEngine()
• StateEngine()
• TODOEngine()

The third engine mirrors the context.TODO() pattern - the "TODO"
engine indicates that work needs to be done to even arrive at an
operation that targets one of the engines specifically; or that there is
something that should be revisited about this particular usage.

With the stability period coming up, we will want to avoid large
mechanical changes in the near future, so now is a good time to get this
out of the way and to work towards a "prototype plus" of the raft log
separation for the remaining weeks.

Ideally this would culminate in a way to actually run with two engines
for experiments (ignoring crash-restart correctness issues and the
like), which would be an important milestone for the project and would
give us a reality check on the work that lies ahead to productionize
this work.

This refactor was carried out mechanically and it assigns everything
to the TODOEngine. In other words, no brains have been put into
deciding which engine to use yet. This will be done piecemeal in
smaller PRs down the line, and will help discover new issues for
the CRDB-220 epic.

Epic: CRDB-220
Release note: None

Footnotes

1. https://github.com/cockroachdb/cockroach/pulls?q=is%3Apr+%22CRDB-220%22+is%3Amerged+created%3A%3C2023-02-08 ↩

[cockroach-teamcity]

This change is 

[tbg]

bors r=pavelkalinnikov

TFTR!

[craig]

Build succeeded:

• Bazel Essential CI (Cockroach)

[sumeerbhola]

Reviewable status: complete! 0 of 0 LGTMs obtained

---

pkg/kv/kvserver/store.go line 463 at r2 (raw file):

	todoEngine storage.Engine
	// logEngine is the engine holding the raft state.
	logEngine storage.Engine

I am very nervous about correctness if callers can directly interact with the storage.Engine interface, versus a more structured interface like in #88606.

Is this just a stop-gap, so that we know which caller needs what kind of functionality?

[pav-kv]

Reviewable status: complete! 0 of 0 LGTMs obtained

pkg/kv/kvserver/store.go line 463 at r2 (raw file):

	todoEngine storage.Engine
	// logEngine is the engine holding the raft state.
	logEngine storage.Engine

I am very nervous about correctness if callers can directly interact with the storage.Engine interface, versus a more structured interface like in #88606.

Is this just a stop-gap, so that we know which caller needs what kind of functionality?

I'm on the same page with this concern. Pebble storage doesn't seem the right abstraction for raft logs, it's too broad. Having two engines like this can be useful for PoC performance gain demo (which, I think, was the intent behind this PR), but to make it correct I would rather be coming from a very restrictive interface fit exactly for the purpose.