Start CCF network node with config_start.json

[bsenthilr]

New to CCF and trying to migrate an existing setup form 1.0.5 => 4.0.16 the latest release.

The start script based on sandbox.sh works fine without any "--config-file" specified. Curated a config_start.json based on a working template.
But CCF node would not start and believe the problem is associated with config_start.json not getting copied into "ccf/workspace/sandbox_0" folder from the logs. Trying to manually shell into container and copying the json to sandbox_0 seems to fix this problem.

Please let me know if this interpretation is right and if this is not a correct way to start?
Also not clear what "infra.remote" means and why its going through that code path with config provided.

Thanks

Snippet of start command, logs:

${CCF_BASE}/bin/sandbox.sh \
    --config-file config_start.json \
    --verbose \
    --host-log-level debug \
    -p ${F_SERVICEHOME}/lib/libtest \
    -e virtual -t virtual \
    --workspace ${F_SERVICEHOME}/workspace \
    --initial-node-cert-validity-days 365 \
    --initial-service-cert-validity-days 365 &

ccf-governor    | Python environment successfully setup
ccf-governor    | 14:30:59.230 | WARNING  | infra.interfaces:from_json:169 - Converted **ip**:443 to **ip** and 443
ccf-governor    | 14:30:59.231 | WARNING  | __main__:run:49 - Virtual mode enabled
ccf-governor    | 14:30:59.231 | INFO     | __main__:run:50 - Starting 1 CCF node...
ccf-governor    | 14:30:59.231 | INFO     | infra.network:_setup_common_folder:487 - Creating common folder: /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.231 | INFO     | infra.proc:ccall:11 - rm -rf /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.234 | INFO     | infra.proc:ccall:11 - mkdir -p /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.238 | INFO     | infra.proc:ccall:11 - cp /opt/ccf_virtual/bin/actions.js /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.244 | INFO     | infra.proc:ccall:11 - cp /opt/ccf_virtual/bin/validate.js /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.252 | INFO     | infra.proc:ccall:11 - cp /opt/ccf_virtual/bin/resolve.js /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.260 | INFO     | infra.proc:ccall:11 - cp /opt/ccf_virtual/bin/apply.js /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.268 | INFO     | infra.proc:ccall:11 - ln -s /tfl/keygenerator.sh /governor/ccf/workspace/sandbox_common
ccf-governor    | 14:30:59.274 | INFO     | infra.proc:ccall:11 - /opt/ccf_virtual/bin/keygenerator.sh --name member0 --curve secp384r1 --gen-enc-key [cwd: /governor/ccf/workspace/sandbox_common]
ccf-governor    | 14:30:59.353 | INFO     | infra.member:__init__:295 - Member member0 created: **

ccf-governor    | 14:30:59.353 | DEBUG    | infra.network:_start_all_nodes:415 - Opening CCF service on [HostSpec(rpc_interfaces={'primary_rpc_interface': RPCInterface(host='**ip**', port=443, protocol='local', transport='tcp', public_host='**ip**', public_port=443, max_open_sessions_soft=1000, max_open_sessions_hard=1010, max_http_body_size=1048576, max_http_header_size=16384, max_http_headers_count=256, max_concurrent_streams_count=100, initial_window_size=65536, max_frame_size=16384, endorsement=Endorsement(authority=<EndorsementAuthority.Service: 1>, acme_configuration=None), acme_configuration=None, accepted_endpoints=None, forwarding_timeout_ms=3000, app_protocol=<AppProtocol.HTTP1: 1>)}, acme_challenge_server_interface=None)]
ccf-governor    | 14:30:59.354 | INFO     | infra.remote:__init__:746 - Node 0: Using configuration file /tfl/governor/etc/config_start.json
ccf-governor    | 14:30:59.354 | INFO     | infra.remote:_rc:392 - [**ip**] rm -rf /governor/ccf/workspace/sandbox_0
ccf-governor    | 14:30:59.362 | INFO     | infra.remote:_rc:392 - [**ip**] mkdir -p /governor/ccf/workspace/sandbox_0
ccf-governor    | 14:30:59.370 | INFO     | infra.remote:_rc:392 - [**ip**] ln -s /opt/ccf_virtual/bin/cchost /governor/ccf/workspace/sandbox_0/cchost
ccf-governor    | 14:30:59.374 | INFO     | infra.remote:_rc:392 - [**ip**] ln -s /governor/ccf/lib/libgovernorenc.virtual.so /governor/ccf/workspace/sandbox_0/libgovernorenc.virtual.so
ccf-governor    | 14:30:59.379 | INFO     | infra.remote:_rc:392 - [**ip**] cp /governor/ccf/workspace/sandbox_common/member0_cert.pem /governor/ccf/workspace/sandbox_0/member0_cert.pem
ccf-governor    | 14:30:59.385 | INFO     | infra.remote:_rc:392 - [**ip**] cp /governor/ccf/workspace/sandbox_common/member0_enc_pubk.pem /governor/ccf/workspace/sandbox_0/member0_enc_pubk.pem
ccf-governor    | 14:30:59.393 | INFO     | infra.remote:_rc:392 - [**ip**] cp /governor/ccf/workspace/sandbox_common/actions.js /governor/ccf/workspace/sandbox_0/actions.js
ccf-governor    | 14:30:59.399 | INFO     | infra.remote:_rc:392 - [**ip**] cp /governor/ccf/workspace/sandbox_common/validate.js /governor/ccf/workspace/sandbox_0/validate.js
ccf-governor    | 14:30:59.405 | INFO     | infra.remote:_rc:392 - [**ip**] cp /governor/ccf/workspace/sandbox_common/resolve.js /governor/ccf/workspace/sandbox_0/resolve.js
ccf-governor    | 14:30:59.414 | INFO     | infra.remote:_rc:392 - [**ip**] cp /governor/ccf/workspace/sandbox_common/apply.js /governor/ccf/workspace/sandbox_0/apply.js
ccf-governor    | 14:30:59.421 | INFO     | infra.remote:start:446 - [**ip**] cd /governor/ccf/workspace/sandbox_0 && ./cchost --config config_start.json --enclave-log-level info --enclave-file ./libgovernorenc.virtual.so 1> /governor/ccf/workspace/sandbox_0/out 2> /governor/ccf/workspace/sandbox_0/err (env: dict_keys(['UBSAN_OPTIONS', 'TSAN_OPTIONS', 'ASAN_OPTIONS']))
ccf-governor    | start_ccf_network.sh: wait for cchost to write the ledger keys
ccf-governor    | 14:31:04.439 | ERROR    | infra.network:_start_all_nodes:469 - Failed to start node 0
ccf
ccf-governor    |   File "/opt/ccf_virtual/bin/infra/network.py", line 606, in start_and_open
ccf-governor    |     self.start(args, **kwargs)
ccf-governor    |     │    │     │       └ {}
ccf-governor    |     │    │     └ Namespace(auto_shutdown=False, auto_shutdown_delay_s=0, binary_dir='/opt/ccf_virtual/bin', ccf_version=None, common_dir=None,...
ccf-governor    |     │    └ <function Network.start at 0x7f24ccc71f70>
ccf-governor    |     └ <infra.network.Network object at 0x7f24d51896a0>
ccf-governor    |
ccf-governor    |   File "/opt/ccf_virtual/bin/infra/network.py", line 559, in start
ccf-governor    |     primary = self._start_all_nodes(args, **kwargs)
ccf-governor    |               │    │                │       └ {}
ccf-governor    |               │    │                └ Namespace(auto_shutdown=False, auto_shutdown_delay_s=0, binary_dir='/opt/ccf_virtual/bin', ccf_version=None, common_dir=None,...
ccf-governor    |               │    └ <function Network._start_all_nodes at 0x7f24ccc71e50>
ccf-governor    |               └ <infra.network.Network object at 0x7f24d51896a0>
ccf-governor    |
ccf-governor    | > File "/opt/ccf_virtual/bin/infra/network.py", line 428, in _start_all_nodes
ccf-governor    |     node.start(
ccf-governor    |     │    └ <function Node.start at 0x7f24ccc64ee0>
ccf-governor    |     └ <infra.node.Node object at 0x7f24d5154fa0>
ccf-governor    |
ccf-governor    |   File "/opt/ccf_virtual/bin/infra/node.py", line 213, in start
ccf-governor    |     self._start(
ccf-governor    |     │    └ <function Node._start at 0x7f24ccc61160>
ccf-governor    |     └ <infra.node.Node object at 0x7f24d5154fa0>
ccf-governor    |
ccf-governor    |   File "/opt/ccf_virtual/bin/infra/node.py", line 341, in _start
ccf-governor    |     raise RuntimeError(
ccf-governor    |
ccf-governor    | RuntimeError: Error starting node 0
ccf-governor    | 14:31:04.445 | INFO     | infra.network:network:1592 - Stopping network
ccf-governor    | 14:31:04.445 | INFO     | infra.node:stop:444 - Stopping node 0
ccf-governor    | 14:31:04.445 | INFO     | infra.remote:stop:512 - [**ip**] closing
ccf-governor    | 14:31:04.445 | ERROR    | infra.remote:stop:528 - **ip** exited with signal: Aborted
ccf-governor    | 14:31:04.445 | ERROR    | infra.network:log_errors:146 - Contents of /governor/ccf/workspace/sandbox_0/err:
ccf-governor    | libc++abi: terminating due to uncaught exception of type std::logic_error: Error parsing configuration file config_start.json: [json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal
ccf-governor    

ccf-governor    | 14:31:04.445 | INFO     | __main__:run:159 - Please raise a bug if the issue is unexpected: https://github.com/microsoft/CCF/issues/new?assignees=&labels=bug&template=bug_report.md&title=Unexpected%20error%20when%20running%20sandbox%20script

[bsenthilr]

This diff takes care of the issue

+++ remote.py	2024-04-30 17:27:01.640000000 +0000
@@ -413,7 +413,7 @@
             if len(path) > 0:
                 dst_path = os.path.join(self.root, os.path.basename(path))
                 self.cp(path, dst_path)
-
+
     def get(
         self,
         src_path,

[achamayou]

@bsenthilr is this a migration for a live system, or for a code base/application? If it's the former, you will need to first upgrade to the last 1.x patch, then to 2.0, then to the last 2.x patch etc. Please see Operations compatibility for detail.

If it's the latter, you will want to look at the releases notes for 2.0, 3.0 and 4.0 for applicable configuration changes.

The --config-file option to the sandbox only allows overriding the default rpc interface choices otherwise generated. The infrastructure must be able to control other aspects of the config such as file paths, and node to node interfaces, in order to configure a network successfully. Overwriting the config may work for trivial single node networks, but is unlikely to be successful in general.

I hope this helps, but if not, please do feel free to ask further questions!

[bsenthilr]

Thanks @achamayou.
Working on upgrading code base so its more flexible.
Thanks for clarifying RPC override flow.
And yes agree regarding infra config.