[v22.3.x] cloud_storage: parallelize remote_segment stop

[andrwng]

Backport of #9207

CONFLICT:

• omits changes to many_partitions_test which doesn't have a tiered storage case on this branch

The eviction loop is currently run serially and is waited for by each remote_partition upon stopping, which effecitvely serializes partition_manager stopping each of its partitions. This resulted in what looked like a hang, but was actually a series of waits to stop cloud segments.

This commit parallelizes segment stopping in both the eviction loop and in the remote_partition stop call. A test that previously took over 9 minutes to shutdown a node now takes 10 seconds.

I also considered moving the eviction loop into the remote_partition to further reduce partitions waiting for each other to complete, but that wasn't necessary to avoid this issue.

This also adds a test case similar to the tiered storage many_partitions_test case that reproduces slow shutdown with a large number of partitions and a large number of segments. On my local workstation, the test consistently fails to finish shutting down in 30 seconds without these changes.

(cherry picked from commit b8ba09c)

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.1.x
• v22.3.x
• v22.2.x

Release Notes

Improvements

• Shutting down a server with many hydrated tiered storage segments and partitions is now significantly faster.

[BenPope]

@andrwng anything blocking this?

[andrwng]

@andrwng anything blocking this?

Thanks for the ping. Green CI and a review from the storage team. I retriggered CI, and will poke some folks once that passes

[BenPope]

@andrwng anything blocking triage of the failures, making this ready for review, and assigning reviewers?

[andrwng]

Nothing's led me to be suspicious of this change specifically, but there are a lot of failures in this branch (over the course of a few retries):

• CI Failure (KgoVerifier failed waiting for worker) in ShadowIndexingManyPartitionsTest.test_many_partitions_shutdown #9544
• fixed [v22.3.x] Add config properties to metrics report #9303
• CI Failure (Can't truncate manifest up to offset ..., offset out of range (via BadLogLines)) in CloudRetentionTest.test_cloud_retention #9286
• [v22.3.x] CI Failure (slightly high HWMs in read replicas) in TestReadReplicaService.test_identical_hwms.partition_count=5 #10179
• [v22.3.x] CI Failure (missing warning log line) in RpkRedpandaStartTest.test_rpc_tls_list #10180
• [v22.3.x] CI Failure (decommissioned node doesn't attempt to join) in NodesDecommissioningTest.test_node_is_not_allowed_to_join_after_restart #10183

[BenPope]

Nothing's led me to be suspicious of this change specifically, but there are a lot of failures in this branch (over the course of a few retries):

Understood. CI failures are probably >50% of the reason I'm doing these reminders. My hope is that it'll settle. 🤞

[piyushredpanda]

CI failure seems #9646 and #10024