Add --existing-hasura-url flag for indexer API in localnet

[banool]

Description

With the CLI as it is right now, it is difficult to run a localnet in a Docker Compose environment given the localnet itself tries to create Docker containers. You can get around this by binding the host Docker daemon socket into the localnet container but it isn't very ergonomic or cross platform. To that end, we add support here for the --existing-hasura-url flag. If this is set, we won't try to run Hasura, but will just trust that Hasura is already running there and we'll apply the metadata after the processors have been started. With this and --use-host-postgres combined it is possible to use --with-indexer-api without the localnet ever trying to run anything with Docker.

How Has This Been Tested?

Normal case

Run a localnet normally, see that it still works:

cargo run -p aptos -- node run-localnet --with-indexer-api --force-restart --assume-yes

Existing Hasura instance

Run a postgres instance:

docker run -it -p 5438:5432 -e POSTGRES_HOST_AUTH_METHOD=trust postgres:15

Create the local_testnet database:

psql -h 127.0.0.1 -p 5438 -U postgres -c "CREATE DATABASE local_testnet;"

Run Hasura locally by hand:

$ cat /tmp/env.env
PG_DATABASE_URL=postgres://[email protected]:5438/local_testnet
HASURA_GRAPHQL_METADATA_DATABASE_URL=postgres://[email protected]:5438/local_testnet
INDEXER_V2_POSTGRES_URL=postgres://[email protected]:5438/local_testnet
HASURA_GRAPHQL_ENABLE_CONSOLE=true
HASURA_GRAPHQL_CONSOLE_ASSETS_DIR=/srv/console-assets

docker run -it -p 8092:8080 --env-file /tmp/env.env hasura/graphql-engine:v2.44.0-ce

Run a localnet with an existing DB and Hasura instance:

cargo run -p aptos -- node run-localnet --with-indexer-api --existing-hasura-url http://127.0.0.1:8092 --use-host-postgres --host-postgres-port 5438

See that the localnet comes up successfully:

Indexer API is ready. Endpoint: http://127.0.0.1:8092/
Postgres is ready. Endpoint: postgres://[email protected]:5438/local_testnet
Node API is ready. Endpoint: http://127.0.0.1:8080/
Transaction stream is ready. Endpoint: http://127.0.0.1:50051/
Faucet is ready. Endpoint: http://127.0.0.1:8081/

Applying post startup steps...

Setup is complete, you can now use the localnet!

When I visit http://127.0.0.1:8092/console I see that the metadata is applied and I can make queries successfully.

Note that you can't use --force-restart when --use-host-postgres and --existing-hasura-url are both set because Hasura is already connected to the DB. We figure this is fine because you should only be using --existing-hasura-url if you're doing something like running the localnet in a Docker Compose setup and you're creating a fresh DB every time anyway.

Skip metadata apply

See that the --skip-metadata-apply flag works:

cargo run -p aptos -- node run-localnet --with-indexer-api --skip-metadata-apply --force-restart --assume-yes

After this the localnet comes up successfully but the Hasura instance is unconfigured.

Key Areas to Review

Ensure the logic is correct, especially confirming that Docker isn't required for the localnet if the appropriate combination of flags is specified.

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 1h 20m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
rust-move-tests	12m	🟩
rust-move-tests	12m	🟩
check-dynamic-deps	12m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-cargo-deny	8m	🟩 🟩 🟩 ⬜ 🟩
rust-doc-tests	5m	🟩
execution-performance / test-target-determinator	4m	🟩
rust-move-tests	4m	⬜
test-target-determinator	4m	🟩
check	4m	🟩
rust-move-tests	4m	⬜
semgrep/ci	3m	🟩 🟩 🟩 🟩 🟩 (+1 more)
general-lints	3m	🟩 🟩 🟩 🟩 🟩
fetch-last-released-docker-image-tag	2m	🟩
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+1 more)
permission-check	29s	🟩 🟩 🟩 🟩 🟩 (+1 more)

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite realistic_env_max_load success on 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b

two traffics test: inner traffic : committed: 13989.39 txn/s, submitted: 13990.33 txn/s, expired: 0.95 txn/s, latency: 2837.85 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 4100 ms), latency samples: 5319140
two traffics test : committed: 100.04 txn/s, latency: 1485.34 ms, (p50: 1400 ms, p70: 1500, p90: 1600 ms, p99: 5200 ms), latency samples: 1740
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.103, avg: 1.636", "ConsensusProposalToOrdered: max: 0.316, avg: 0.298", "ConsensusOrderedToCommit: max: 0.399, avg: 0.386", "ConsensusProposalToCommit: max: 0.696, avg: 0.684"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.60s no progress at version 54066 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.88s no progress at version 2563284 (avg 8.88s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b

Compatibility test results for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b (PR)
Upgrade the nodes to version: 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1329.86 txn/s, submitted: 1334.88 txn/s, failed submission: 5.02 txn/s, expired: 5.02 txn/s, latency: 2388.66 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5100 ms), latency samples: 111260
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1290.03 txn/s, submitted: 1292.96 txn/s, failed submission: 2.94 txn/s, expired: 2.94 txn/s, latency: 2394.96 ms, (p50: 2100 ms, p70: 2400, p90: 3700 ms, p99: 5100 ms), latency samples: 114200
5. check swarm health
Compatibility test for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b passed
Upgrade the remaining nodes to version: 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1215.41 txn/s, submitted: 1219.63 txn/s, failed submission: 4.22 txn/s, expired: 4.22 txn/s, latency: 2661.05 ms, (p50: 2300 ms, p70: 2900, p90: 4700 ms, p99: 6500 ms), latency samples: 109500
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite compat success on be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b

Compatibility test results for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b (PR)
1. Check liveness of validators at old version: be3ccc58ef518259d63fab7f3e3613f2a6214268
compatibility::simple-validator-upgrade::liveness-check : committed: 13738.93 txn/s, latency: 2468.48 ms, (p50: 2000 ms, p70: 2200, p90: 3400 ms, p99: 8800 ms), latency samples: 442260
2. Upgrading first Validator to new version: 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6726.49 txn/s, latency: 4245.69 ms, (p50: 5000 ms, p70: 5100, p90: 5300 ms, p99: 5400 ms), latency samples: 121280
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6220.43 txn/s, latency: 5184.06 ms, (p50: 5200 ms, p70: 5300, p90: 7400 ms, p99: 7800 ms), latency samples: 215980
3. Upgrading rest of first batch to new version: 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6636.59 txn/s, latency: 4215.70 ms, (p50: 4900 ms, p70: 5100, p90: 5200 ms, p99: 5300 ms), latency samples: 122760
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6771.82 txn/s, latency: 4343.97 ms, (p50: 4500 ms, p70: 4600, p90: 4700 ms, p99: 5200 ms), latency samples: 250680
4. upgrading second batch to new version: 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10621.10 txn/s, latency: 2553.84 ms, (p50: 2400 ms, p70: 2800, p90: 3500 ms, p99: 3700 ms), latency samples: 190820
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10515.74 txn/s, latency: 2924.44 ms, (p50: 2400 ms, p70: 3400, p90: 4900 ms, p99: 6000 ms), latency samples: 345860
5. check swarm health
Compatibility test for be3ccc58ef518259d63fab7f3e3613f2a6214268 ==> 18660211c6403d4afcaf1bf4f9d10f9fe6be4e9b passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking