feat: Provide easy prover setup

prover/docs/05_proving_batch.md

@@ -0,0 +1,126 @@
+# Proving a batch
+
+If you got to this section, then most likely you are wondering how to prove and verify the batch by yourself. After
+releases `prover-v15.1.0` and `core-v24.9.0` prover subsystem doesn't need access to core database anymore, which means
+you can run only prover subsystem and prove batches without running the whole core system. This guide will help you with
+that.
+
+## Requirements
+
+First of all, you need to install CUDA drivers, all other things will be dealt with by `zk_inception` tool. For that,
+check the following [guide](./02_setup.md)(you can skip bellman-cuda step).
+
+Now, you can use `zk_inception` tool for setting up the env and running prover subsystem. Check the steps for
+installation in [this guide](../../zk_toolbox/README.md). And don't forget to install the prerequisites!
+
+## Initializing system
+
+After you have installed the tool, you can run the prover subsystem by running:
+
+```shell
+zk_inception ecosystem create
+```
+
+The command will create the ecosystem and all the necessary components for the prover subsystem. Now, you need to
+initialize the prover subsystem by running:
+
+```shell
+zk_inception prover init
+```
+
+For the prompts, you need to put `Yes` for copying the configs and setting up the database.
+
+## Proving the batch
+
+### Getting data needed for proving
+
+At this step, we need to get the witness inputs data for the batch you want to prove. Database information now lives in
+input file, called `witness_inputs_<batch>.bin` generated by different core components).
+
+- If batch was produced by your system, the file is stored by prover gateway in GCS (or your choice of object storage --
+  check config). To access it from GCS (assuming you have access to the bucket), run:
+
+  ```shell
+  gsutil cp gs://your_bucket/witness_inputs/witness_inputs_<batch>.bin <path/to/era/prover/artifacts/witness_inputs>
+  ```
+
+- If you want to prove the batch produced by zkSync, you can get the data from the `ExternalProofIntegrationAPI` using
+  `{address}/proof_generation_data` endpoint. You need to replace `{address}` with the address of the API and provide
+  the batch number as a query data to get the data for specific batch, otherwise, you will receive latest data for the
+  batch, that was already proven.
+
+### Preparing database
+
+After you have the data, you need to prepare the system to run the batch. So, database needs to know about the batch and
+the protocol version it should use. Check the latest protocol version in the codebase by checking const
+`PROVER_PROTOCOL_SEMANTIC_VERSION` or run the binary in `prover` workspace:
+
+```console
+cargo run --bin prover_version
+```
+
+It will give you the latest prover protocol version in a semver format, like `0.24.2`, you need to know only minor and
+patch versions. Now, go to the `prover/crates/bin/vk_setup_data_generator_server_fri/data/commitments.json` and get
+`snark_wrapper` value from it. Then, you need to insert the info about protocol version into the database. First,
+connect to the database, e.g. locally you can do it like that(for local DB you can find the url in `prover.yaml` or
+`general.yaml` files):
+
+```shell
+psql postgres://postgres:notsecurepassword@localhost/prover_local
+```
+
+And run the following query:
+
+```shell
+INSERT INTO
+prover_fri_protocol_versions (
+id,
+recursion_scheduler_level_vk_hash,
+created_at,
+protocol_version_patch
+)
+VALUES
+(<minor version>, '<snark wrapper value>'::bytea, NOW(), <patch version>)
+ON CONFLICT (id, protocol_version_patch) DO NOTHING
+
+```
+
+Now, you need to insert the batch into the database. Run the following query:
+
+```shell
+INSERT INTO
+witness_inputs_fri (
+l1_batch_number,
+witness_inputs_blob_url,
+protocol_version,
+status,
+created_at,
+updated_at,
+protocol_version_patch
+)
+VALUES
+(<batch number>, 'witness_inputs_<batch_number>.bin', <minor version>, 'queued', NOW(), NOW(), <patch version>)
+ON CONFLICT (l1_batch_number) DO NOTHING
+```
+
+## Running prover subsystem
+
+At this step, all the data is prepared and you can run the prover subsystem. To do that, run the following commands:
+
+```shell
+zk_inception prover run --component=prover
+zk_inception prover run --component=witness-generator --round=all-rounds
+zk_inception prover run --component=witness-vector-generator --threads=10
+zk_inception prover run --component=compressor
+```
+
+And you are good to go! The prover subsystem will prove the batch and you can check the results in the database.
+
+## Verifying zkSync batch
+
+Now, assuming the proof is already generated, you can verify using `ExternalProofIntegrationAPI`. Usually proof is
+stored in GCS bucket(for which you can use the same steps as for getting the witness inputs data
+[here](#getting-data-needed-for-proving), but locally you can find it in `/artifacts/proofs_fri` directory). Now, simply
+send the data to the endpoint `{address}/verify_batch/{batch_number}`. Note, that you need to pass the generated proof
+as serialized JSON data when calling the endpoint. API will respond with status 200 if the proof is valid and with the
+error message otherwise.

zk_toolbox/crates/config/src/secrets.rs

@@ -26,6 +26,18 @@ pub fn set_databases(
     Ok(())
 }
 
+pub fn set_prover_database(
+    secrets: &mut SecretsConfig,
+    prover_db_config: &DatabaseConfig,
+) -> anyhow::Result<()> {
+    let database = secrets
+        .database
+        .as_mut()
+        .context("Databases must be presented")?;
+    database.prover_url = Some(SensitiveUrl::from(prover_db_config.full_url()));
+    Ok(())
+}
+
 pub fn set_l1_rpc_url(secrets: &mut SecretsConfig, l1_rpc_url: String) -> anyhow::Result<()> {
     secrets
         .l1

zk_toolbox/crates/zk_inception/README.md

@@ -269,6 +269,7 @@ Initialize chain, deploying necessary contracts and performing on-chain operatio
 - `-u`, `--use-default` — Use default database urls and names
 - `-d`, `--dont-drop`
 - `--deploy-paymaster <DEPLOY_PAYMASTER>`
+- `--copy-configs` - Copy default configs to the chain
 
   Possible values: `true`, `false`
 
@@ -418,7 +419,7 @@ Initialize prover
 - `--project-id <PROJECT_ID>`
 - `--shall-save-to-public-bucket <SHALL_SAVE_TO_PUBLIC_BUCKET>`
 
-  Possible values: `true`, `false`
+Possible values: `true`, `false`
 
 - `--public-store-dir <PUBLIC_STORE_DIR>`
 - `--public-bucket-base-url <PUBLIC_BUCKET_BASE_URL>`
@@ -428,8 +429,10 @@ Initialize prover
 - `--public-project-id <PUBLIC_PROJECT_ID>`
 - `--bellman-cuda-dir <BELLMAN_CUDA_DIR>`
 - `--download-key <DOWNLOAD_KEY>`
+- `--setup-database`
+- `--copy-configs`
 
-  Possible values: `true`, `false`
+Possible values: `true`, `false`
 
 - `--setup-key-path <SETUP_KEY_PATH>`
 - `--cloud-type <CLOUD_TYPE>`

zk_toolbox/crates/zk_inception/src/commands/prover/args/init.rs

@@ -1,23 +1,29 @@
 use clap::{Parser, ValueEnum};
-use common::{logger, Prompt, PromptConfirm, PromptSelect};
+use common::{db::DatabaseConfig, logger, Prompt, PromptConfirm, PromptSelect};
+use config::ChainConfig;
 use serde::{Deserialize, Serialize};
+use slugify_rs::slugify;
 use strum::{EnumIter, IntoEnumIterator};
+use url::Url;
 use xshell::Shell;
 use zksync_config::configs::fri_prover::CloudConnectionMode;
 
 use super::init_bellman_cuda::InitBellmanCudaArgs;
 use crate::{
     commands::prover::gcs::get_project_ids,
     consts::{DEFAULT_CREDENTIALS_FILE, DEFAULT_PROOF_STORE_DIR},
+    defaults::{generate_db_names, DBNames, DATABASE_PROVER_URL},
     messages::{
-        MSG_CLOUD_TYPE_PROMPT, MSG_CREATE_GCS_BUCKET_LOCATION_PROMPT,
+        msg_prover_db_name_prompt, msg_prover_db_url_prompt, MSG_CLOUD_TYPE_PROMPT,
+        MSG_COPY_CONFIGS_PROMPT, MSG_CREATE_GCS_BUCKET_LOCATION_PROMPT,
         MSG_CREATE_GCS_BUCKET_NAME_PROMTP, MSG_CREATE_GCS_BUCKET_PROJECT_ID_NO_PROJECTS_PROMPT,
         MSG_CREATE_GCS_BUCKET_PROJECT_ID_PROMPT, MSG_CREATE_GCS_BUCKET_PROMPT,
         MSG_DOWNLOAD_SETUP_KEY_PROMPT, MSG_GETTING_PROOF_STORE_CONFIG,
         MSG_GETTING_PUBLIC_STORE_CONFIG, MSG_PROOF_STORE_CONFIG_PROMPT, MSG_PROOF_STORE_DIR_PROMPT,
         MSG_PROOF_STORE_GCS_BUCKET_BASE_URL_ERR, MSG_PROOF_STORE_GCS_BUCKET_BASE_URL_PROMPT,
-        MSG_PROOF_STORE_GCS_CREDENTIALS_FILE_PROMPT, MSG_SAVE_TO_PUBLIC_BUCKET_PROMPT,
-        MSG_SETUP_KEY_PATH_PROMPT,
+        MSG_PROOF_STORE_GCS_CREDENTIALS_FILE_PROMPT, MSG_PROVER_DB_NAME_HELP,
+        MSG_PROVER_DB_URL_HELP, MSG_SAVE_TO_PUBLIC_BUCKET_PROMPT, MSG_SETUP_KEY_PATH_PROMPT,
+        MSG_USE_DEFAULT_DATABASES_HELP,
     },
 };
 
@@ -54,6 +60,20 @@ pub struct ProverInitArgs {
     #[serde(flatten)]
     pub setup_key_config: SetupKeyConfigTmp,
 
+    #[clap(long)]
+    pub copy_configs: Option<bool>,
+
+    #[clap(long)]
+    pub setup_database: Option<bool>,
+    #[clap(long, help = MSG_PROVER_DB_URL_HELP)]
+    pub prover_db_url: Option<Url>,
+    #[clap(long, help = MSG_PROVER_DB_NAME_HELP)]
+    pub prover_db_name: Option<String>,
+    #[clap(long, short, help = MSG_USE_DEFAULT_DATABASES_HELP)]
+    pub use_default: Option<bool>,
+    #[clap(long, short, action)]
+    pub dont_drop: Option<bool>,
+
     #[clap(long)]
     cloud_type: Option<InternalCloudConnectionMode>,
 }
@@ -160,33 +180,46 @@ pub struct SetupKeyConfig {
     pub setup_key_path: String,
 }
 
+#[derive(Debug, Clone)]
+pub struct ProverDatabaseConfig {
+    pub database_config: DatabaseConfig,
+    pub dont_drop: bool,
+}
+
 #[derive(Debug, Clone)]
 pub struct ProverInitArgsFinal {
     pub proof_store: ProofStorageConfig,
     pub public_store: Option<ProofStorageConfig>,
     pub setup_key_config: SetupKeyConfig,
     pub bellman_cuda_config: InitBellmanCudaArgs,
     pub cloud_type: CloudConnectionMode,
+    pub copy_configs: bool,
+    pub database_config: Option<ProverDatabaseConfig>,
 }
 
 impl ProverInitArgs {
     pub(crate) fn fill_values_with_prompt(
         &self,
         shell: &Shell,
         setup_key_path: &str,
+        chain_config: &ChainConfig,
     ) -> anyhow::Result<ProverInitArgsFinal> {
         let proof_store = self.fill_proof_storage_values_with_prompt(shell)?;
         let public_store = self.fill_public_storage_values_with_prompt(shell)?;
         let setup_key_config = self.fill_setup_key_values_with_prompt(setup_key_path);
         let bellman_cuda_config = self.fill_bellman_cuda_values_with_prompt()?;
         let cloud_type = self.get_cloud_type_with_prompt();
+        let copy_configs = self.ask_copy_configs();
+        let database_config = self.fill_database_values_with_prompt(chain_config);
 
         Ok(ProverInitArgsFinal {
             proof_store,
             public_store,
             setup_key_config,
             bellman_cuda_config,
             cloud_type,
+            copy_configs,
+            database_config,
         })
     }
 
@@ -440,4 +473,62 @@ impl ProverInitArgs {
 
         cloud_type.into()
     }
+
+    fn ask_copy_configs(&self) -> bool {
+        self.copy_configs
+            .unwrap_or_else(|| PromptConfirm::new(MSG_COPY_CONFIGS_PROMPT).ask())
+    }
+
+    fn fill_database_values_with_prompt(
+        &self,
+        config: &ChainConfig,
+    ) -> Option<ProverDatabaseConfig> {
+        let setup_database = self
+            .setup_database
+            .clone()
+            .unwrap_or_else(|| PromptConfirm::new("Do you want to setup the database?").ask());
+
+        if setup_database {
+            let DBNames { prover_name, .. } = generate_db_names(config);
+            let chain_name = config.name.clone();
+
+            let dont_drop = self.dont_drop.clone().unwrap_or_else(|| {
+                !PromptConfirm::new("Do you want to drop the database?")
+                    .default(true)
+                    .ask()
+            });
+
+            if self.use_default.clone().unwrap_or_else(|| {
+                PromptConfirm::new(MSG_USE_DEFAULT_DATABASES_HELP)
+                    .default(true)
+                    .ask()
+            }) {
+                Some(ProverDatabaseConfig {
+                    database_config: DatabaseConfig::new(DATABASE_PROVER_URL.clone(), prover_name),
+                    dont_drop,
+                })
+            } else {
+                let prover_db_url = self.prover_db_url.clone().unwrap_or_else(|| {
+                    Prompt::new(&msg_prover_db_url_prompt(&chain_name))
+                        .default(DATABASE_PROVER_URL.as_str())
+                        .ask()
+                });
+
+                let prover_db_name: String = self.prover_db_name.clone().unwrap_or_else(|| {
+                    Prompt::new(&msg_prover_db_name_prompt(&chain_name))
+                        .default(&prover_name)
+                        .ask()
+                });
+
+                let prover_db_name = slugify!(&prover_db_name, separator = "_");
+
+                Some(ProverDatabaseConfig {
+                    database_config: DatabaseConfig::new(prover_db_url, prover_db_name),
+                    dont_drop,
+                })
+            }
+        } else {
+            None
+        }
+    }
 }