Output all combined keystores to a single directory

[OisinKyne]

🎯 Problem to be solved

We currently combine keys and write the keystores to subdirectories of the input folder given to charon combine. This structure is not friendly for importing into validator clients.

🛠️ Proposed solution

• Approved design doc: link

• Core team consensus on the proposed solution

• Make an optional flag --output-dir (default: "validator_keys") that indicates the folder that keystores from charon combine are written to.

Here is the problem where this ticket arose from #1799 (comment)

🧪 Tests

• Tested by new automated unit/integration/smoke tests

👐 Additional acceptance criteria

• Update the docs cli reference to match this new output structure.

❌ Out of Scope

If there is anything to highlight as out of scope for this issue, please outline it here.

[gsora]

Right now the combine command outputs the following directories:

$ tree ./validators-to-be-combined

validators-to-be-combined/
├── 0x822c5310674f4fc4ec595642d0eab73d01c62b588f467da6f98564f292a975a0ac4c3a10f1b3a00ccc166a28093c2dcd
│   └── validator_keys
│       ├── keystore-0.json
│       └── keystore-0.txt
├── 0x8929b4c8af2d2eb222d377cac2aa7be950e71d2b247507d19b5fdec838f0fb045ea8910075f191fd468da4be29690106
│   └── validator_keys
│       ├── keystore-0.json
│       └── keystore-0.txt

that is, for each validator defined in the lock file one directory is created named after its public key and the private keys are placed in a subdirectory, called validator_keys.

This format is compatible with Charon and should be compatible with any other validator client that supports the Ethereum keystore format.

So @OisinKyne what you're suggesting is that instead of employing the directory structure I outlined earlier we do this:

$ tree ./validators-to-be-combined

validators-to-be-combined/
└── validator_keys
    ├── keystore-0.json
    ├── keystore-0.txt
    ├── keystore-1.json
    └── keystore-1.txt

Correct?

[OisinKyne]

So @OisinKyne what you're suggesting is that instead of employing the directory structure I outlined earlier we do this:

$ tree ./validators-to-be-combined

validators-to-be-combined/
└── validator_keys
    ├── keystore-0.json
    ├── keystore-0.txt
    ├── keystore-1.json
    └── keystore-1.txt

Correct?

Yes but maybe we take an explicit (but with a default) output path, rather than placing the generated folder into the input folder, for fear of collisions or confusion or something.

Usage:
  charon combine [flags]

Flags:
      --cluster-dir string   Parent directory containing a number of .charon subdirectories from each node in the cluster. (default ".charon/cluster/")
      --force                Overwrites private keys with the same name if present.
  -h, --help                 Help for combine
       --output-dir string   Directory to output the combined private keys to. (default ./validator_keys/)

so if I made a folder ./distributed-validator-nodes-to-combine/, dumped all the .charon folders into it, and then from ./ I ran: charon combine --cluster-dir ./distributed-validator-nodes-to-combine/, it would spit out:

tree .
.
├── distributed-validator-nodes-to-combine
│   ├── node0
│   │   ├── charon-enr-private-key
│   │   ├── cluster-lock.json
│   │   ├── deposit-data.json
│   │   └── validator_keys
│   │       ├── keystore-0.json
│   │       └── keystore-0.txt
│   ├── node1
│   │   ├── charon-enr-private-key
│   │   ├── cluster-lock.json
│   │   ├── deposit-data.json
│   │   └── validator_keys
│   │       ├── keystore-0.json
│   │       └── keystore-0.txt
│   ├── node2
│   │   ├── charon-enr-private-key
│   │   ├── cluster-lock.json
│   │   ├── deposit-data.json
│   │   └── validator_keys
│   │       ├── keystore-0.json
│   │       └── keystore-0.txt
│   └── node3
│       ├── charon-enr-private-key
│       ├── cluster-lock.json
│       ├── deposit-data.json
│       └── validator_keys
│           ├── keystore-0.json
│           └── keystore-0.txt
└── validator_keys
    ├── keystore-0.json
    └── keystore-0.txt

11 directories, 22 files

While you are doing this, can you make the default input directory .charon/cluster/, because that is the default output dir of charon create cluster for the time being. Thanks.

[gsora]

Implemented, lmk if it works for you!

[OisinKyne]

Suggested a variable change, once made all good to merge and close. (Though would be good to user test this ideally. Think eridian already made a guide).

[gsora]

I'll go ahead, merge your suggestions and then merge in main.

Thanks!