delete pod to stop a node instead of scaling sts

[aluon]

Description

Add a method to temporarily stop a fullnode/validator node by repeatedly deleting the pod. This is used for the fullnode/validator stress tests where we want to keep to keep the underlying node allocated to the StatefulSet. We suspect that the previous method of scaling the StatefulSet was causing node allocation delays and causing these test to timeout

How Has This Been Tested?

Ran the adhoc forge workflow

• https://github.com/aptos-labs/aptos-core/actions/runs/11710523889
• https://github.com/aptos-labs/aptos-core/actions/runs/11710511768

Key Areas to Review

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 1h 42m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
test-target-determinator	29m	🟩 🟩 ⬜ 🟩 🟩 (+2 more)
adhoc-forge-test / forge	13m	🟩
adhoc-forge-test / forge	13m	🟥
rust-cargo-deny	11m	🟩 🟩 🟩 🟩 🟥 (+2 more)
check-dynamic-deps	10m	🟩 🟩 🟩 🟩 🟩 (+3 more)
semgrep/ci	4m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-move-tests	3m	🟩
general-lints	3m	🟩 🟩 🟩 🟩 🟩 (+2 more)
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+2 more)

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[ibalajiarun]

Waiting till healthy will make the node catch up to latest state, which is something we don't want to do. Can we wait till the pod is running instead?

[aluon]

Doesn't that match the existing behavior? These tests were using Node::start() which calls wait_until_healthy()

I updated this to check the pod status instead

[ibalajiarun]

No, before we had stop() -> sleep -> start() -> healthy. Now, kill() -> healthy() -> kill() -> healthy() -> ... right?

[aluon]

Looks like these tests are still timing out with these changes. I'm going to try increasing the timeout

• https://github.com/aptos-labs/aptos-core/actions/runs/11747636025
• https://github.com/aptos-labs/aptos-core/actions/runs/11747636025

[aluon]

This didn't work. We're still seeing delays in pod startup due to reattaching PVCs

I increased the timeouts in #15244 to get the Forge tests working again. We can explore using node affinities to avoid pods getting moved between nodes