node wise recovery

[bharathv]

PRD: link

Given an input set of nodes, generates a move plan that force reconfigures all partitions that lost majority (or all replicas) due to the unavailability of these nodes. This is intended to be used as an escape hatch to bulk move all such majority lost partitions when nodes are dead and are irrecoverable.

Adds the following admin end points. Input for both is array of node_ids.

• GET /v1/partitions/majority_lost?defunct_nodes=<csv nodes> - Given an input set of defunct nodes, generates a list of partitions that should be force recovered (aka lost majority).

• POST /v1/partitions/force_recover_from_nodes - Submits the plan generated from above step and the partitions are eventually force recovered.

Example invocations

curl -X GET http://localhost:9644/v1/partitions/majority_lost\?defunct_nodes\=0,1 

curl -X POST http://localhost:9644/v1/partitions/force_recover_from_nodes -H 'Content-Type: application/json' --data '{"defunct_nodes": [1], "partitions_to_force_recover": [{"ntp": {"ns": "kafka", "topic": "bar", "partition": 0}, "topic_revision": 26, "replicas": [{"node_id": 1, "core": 0}], "defunct_nodes": [1]}]}

partitions_to_force_recover part of the POST payload is the plan generated from GET. The plan is validated before generating the controller command that orchestrates the moves.

Monitoring: Balanacer status (v1/cluster/partition_balancer/status) now contains the following

• partitions_pending_force_recovery_count - # of partitions that are yet to be force recovered.
• partitions_pending_force_recovery_sample - A sample list of partitions pending force recovery (limit capped to 10)

Notes on defunct nodes:

• This PR adds the concept of "defunct nodes", nodes that are not functional and explicitly marked defunct by the user.
• Defunct nodes are generated from force_replicas_from_nodes commands, all the brokers marked as "defunct" in the request are marked defunct for future processing and this is irreversible for now.
• defunct nodes are considered unavailable from RP's perspective. This means replicas from these nodes are moved away by the balancer eventually and any list of user approved force recoverable partitions in the request are force reconfigured.
• liveness status of a broker (functional or defunct) can be checked in the /brokers end point.

Fixes: https://github.com/redpanda-data/planning/issues/84

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

Features

• Adds node wise partition recovery functionality. Given an input set of node ids, force reconfigures all partitions that would lose majority if the input set of nodes are dead. Intended to bulk recover partitions that are stuck when majority of brokers hosting their replicas are dead and irrecoverable.

[vbotbuildovich]

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/38327#018afe5f-7762-4503-a150-c716cc166f9a

[vbotbuildovich]

ducktape was retried in job https://buildkite.com/redpanda/redpanda/builds/38327#018afe6e-270d-4d05-af8b-98baefa51760

[twmb]

GET with a request body is questionable in HTTP, "A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request."

Rather than trying to use methods to determine a dry run, what about using a query parameter?
Ref: https://stackoverflow.com/questions/58831345/how-would-you-design-a-rest-api-that-includes-the-simulation-of-a-post and the linked k8s doc, which uses a dryRun QP

Also,
"""
partitions with at least one surviving replica are reconfigured/downsized to surviving replica set. ex: if 1 of 3 replica remains, the new target assignment only consists of 1 replica
"""
Does this mean the RF for the partition is changed? I feel like we should separate changing the RF from forceful-move-away-from-node.

Also, does this move all partitions or only unhealthy ones?

[bharathv]

Temporarily moving to draft while I polish and get tests passing with the new upcoming revision.

[bharathv]

GET with a request body is questionable in HTTP, "A payload within a GET request message has no defined semantics; sending a payload body on a GET request might cause some existing implementations to reject the request."

Rather than trying to use methods to determine a dry run, what about using a query parameter?
Ref: https://stackoverflow.com/questions/58831345/how-would-you-design-a-rest-api-that-includes-the-simulation-of-a-post and the linked k8s doc, which uses a dryRun QP

Thanks, TIL.. thats a bummer.. I tend to agree with elastic's sentiment on this, let me figure out a way to encode the list of broker ids into a query parameter.

Does this mean the RF for the partition is changed? I feel like we should separate changing the RF from forceful-move-away-from-node.

Any particular reason? In a way we are preserving existing RF so the user doesn't have to run a separate command to up-replicate again.

Also, does this move all partitions or only unhealthy ones?

both (in the upcoming rev). The nodes in the list are marked "defunct" from RP perspective and the balancer tries to move replicas away gracefully (for those that have the majority) and forcefully reconfigure the rest (after a user approval).

[twmb]

In a way we are preserving existing RF
But doesn't the PR description say that if we can't preserve the original RF, then the RF is automatically downsized?

[bharathv]

failures are unrelated (changed some RPC structs, need to update compat test, will do it along with next rev), ready for review.

[bharathv]

Rebased and force pushed to fix conflicts.

[mmaslankaprv]

plase add a it.check() here