storage: use SetOptions() when reusing Pebble iterators

[erikgrinaker]

Iterator reuse now relies on Pebble.SetOptions() to configure the
reused Pebble iterator. This allows a wider range of iterators to be
reused, since previously only the bounds could be changed on existing
iterators.

Release note: None

[cockroach-teamcity]

This change is 

[jbowens]

Heads up that while cockroachdb/pebble#1675 merged, the Pebble bump is blocked on resolving a breaking code change to vfs.WithDiskHealthChecks, which in turn is blocked on cockroachdb/pebble#1690. 😵

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @nicktrav and @sumeerbhola)

[erikgrinaker]

This should be good to go now, PTAL.

A quick kv95 run shows a slight performance regression here. I think we can merge this anyway, and work on performance optimizations later:

master:

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
  180.0s        0       19341212       107450.5      1.6      1.1      4.5     10.5    159.4  read

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
  180.0s        0        1017946         5655.2      4.0      3.3      8.9     15.2    121.6  write

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
  180.0s        0       20359158       113105.7      1.7      1.2      5.0     11.0    159.4  

setoptions:

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
  180.0s        0       18849240       104717.1      1.6      1.2      4.7     10.5    117.4  read

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__total
  180.0s        0         992337         5512.9      4.1      3.4      9.4     15.7    201.3  write

_elapsed___errors_____ops(total)___ops/sec(cum)__avg(ms)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)__result
  180.0s        0       19841577       110230.1      1.7      1.2      5.2     11.0    201.3  

[jbowens]

I think we can merge this anyway, and work on performance optimizations later

Sounds good. I'll look at cockroachdb/pebble#1709 this week, which I think might make a difference here.

Reviewed 4 of 5 files at r4, 4 of 14 files at r5, all commit messages.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @erikgrinaker, @nicktrav, and @sumeerbhola)

---

pkg/storage/pebble_iterator.go line 117 at r5 (raw file):

		} else {
			var err error
			if p.iter, err = iterToClone.Clone(); err != nil {

Just a fyi, I filed cockroachdb/pebble#1712 so that we can avoid the SetOptions call in this case.

[erikgrinaker]

Sounds good, TFTR!

bors r=jbowens

[craig]

Build succeeded:

• GitHub CI (Cockroach)