[CLI] Integrate sui-test-validator into sui start

[stefan-mysten]

Description

This PR integrates sui-test-validator binary into sui start, but keeps the crate for now before being deprecated.

• the default behaviour of the existing sui start is preserved
• the default sui-test-validator behaviour can be achieved by passing two flags: sui start --force-regenesis --with-faucet.
• adds a sui-test-validator.sh bash script that is executable to keep backward compatibility with the previous binary
• keeps the sui-test-validator binary for now.
• changed default port for GraphQL to 9125

One difference in flag/options usage is that sui-test-validator used to start graphql service when the graphql-port arg was passed. To keep it consistent across all three services, they each have their own flag to use to start the service with default host and port, and a port can be provided to it as well:

• --with-faucet
• --with-indexer or --with-indexer=9124
• --with-graphql or --with-graphql=9125

Previously, sui-test-validator exposed a few flags and options (see orig code here), which were migrated to sui start. There are two new flags in sui start, --with-graphql and --with-faucet, to make it easier to start these services with default values. Due to this, the bash script checks for multiple flags and their values and tries to ensure everything is passed correctly.

sui-test-validator implemented a simple faucet wrapper of the sui-faucet crate. This was moved into faucet.rs module in sui crate.

Given that there are three extra services that can be started, plus a configuration variable to start the network without a fullnode, here is an explanation of what each service requires:
graphql requires indexer - without an indexer, the GraphQL will have no data to serve. This will throw an error.
indexer requires fullnode - without a fullnode, the indexer will start ingesting mainnet data with default settings. The program throws an error here.

Test plan

Existing tests.

---

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

• Protocol:
• Nodes (Validators and Full nodes):
• Indexer:
• JSON-RPC:
• GraphQL:
• CLI: Integrated sui-test-validator into sui start.

sui start can now be used to also start an indexer, a GraphQL service, or a faucet. Note that you need to build sui with --features indexer, and have installed libpq to start the indexer and GraphQL services. By default, this feature is turned off to preserve backward compatibility of sui start. Alternatively, you can download the latest release and use the sui-pg binary, which is built using the indexer feature.

• the default behaviour of the existing sui start is preserved
• the default sui-test-validator behaviour can be achieved by passing two flags: sui start --force-regenesis --with-faucet.
• adds a sui-test-validator.sh bash script to sui/scripts folder that is executable to keep backward compatibility with the previous binary
• Rust SDK:

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 26, 2024 11:07pm

[vercel]

@stefan-mysten is attempting to deploy a commit to the Mysten Labs Team on Vercel.

A member of the Team first needs to authorize it.

[ronny-mysten]

Comments on content part. Looks good!

[awelc]

It looks like this PR does a lot more things than what is stated in the description (e.g., all the new options added, lot's of changes in faucet.rs). I would be great to get a bit more of a high-level description on what the implemented/modified features are

[stefan-mysten]

It looks like this PR does a lot more things than what is stated in the description (e.g., all the new options added, lot's of changes in faucet.rs). I would be great to get a bit more of a high-level description on what the implemented/modified features are

It looks like this PR does a lot more things than what is stated in the description (e.g., all the new options added, lot's of changes in faucet.rs). I would be great to get a bit more of a high-level description on what the implemented/modified features are

Good point Adam. I edited the top description. Just to also answer here, the new options added are actually migrated from sui-test-validator, and there are only two new flags: --with-faucet and --with-graphql. More details in the description.
I have not added any new features per-say, as the main goal was to keep compatibility with the commands as they are used today to not disrupt users.

[stefan-mysten]

@johnjmartin @ebmifa we need to ensure two things in the builds :

• the sui binary is not built and released with --features indexer or --all-features. From what I can tell, it's only in one place the binaries are built with cargo build --all-features, but those do not seem to be uploaded anywhere... I think. Could you please confirm that?
• we need to build an additional binary and upload it to our releases, called sui-pg, which is built using cargo build --release --bin sui --features indexer, which has a hard dependency on libpq.

I think the changes here are sufficient, but would appreciate another pair of eyes.
https://github.com/MystenLabs/sui/pull/18204/files#diff-87db21a973eed4fef5f32b267aa60fcee5cbdf03c67fafdc2a9b553bb0b15f34

[amnn]

Any worries regarding cherry picking this to devnet and testnet branches?

I think we spoke about this offline, but we can't pick to these specific branches, because they track positions on existing release branches. So instead, we would need to pick to those release branches, but technically speaking, if you do that and you update the corresponding devnet/testnet tracking branch, you would be telling validators that they need to update their binaries.

[stefan-mysten]

Any worries regarding cherry picking this to devnet and testnet branches?

I think we spoke about this offline, but we can't pick to these specific branches, because they track positions on existing release branches. So instead, we would need to pick to those release branches, but technically speaking, if you do that and you update the corresponding devnet/testnet tracking branch, you would be telling validators that they need to update their binaries.

Yes, I clarified with PE and have a plan. This will be cherry picked into v1.28-release, and when next releases happen Mon/Tue, we will finalize the rest of the needed changes.