Error building CCF with ninja

[anirbanc3]

Describe the bug
I am trying to build CCF from source. I am using the docker ghcr.io/microsoft/ccf/ci/default:build-25-07-2024. I have installed open-enclave sdk inside the docker. Further, I am using ccf-5.0.6 release for the build. I am getting the following error when running ninja

root@0e857000726d:~/CCF/build# ninja
[538/720] Generating signing_key.pem
Generating RSA private key, 3072 bit long modulus (2 primes)
..................................................++++*......++++*..............................................................++++
......++++
e is 3 (0x03)
[539/720] Generating code from EDL, and renaming to .cpp
Generating edge routine, for the Open Enclave SDK.
Processing /root/CCF/edl/ccf.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/platform.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/attestation.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/cpu.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/debug.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/thread.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/switchless.edl.
Processing /opt/openenclave/include/openenclave/edl/sgx/tdx_verification.edl.
Processing /opt/openenclave/include/openenclave/edl/time.edl.
Success.
[544/720] Building CXX object CMakeFiles/js_generic.enclave.dir/src/apps/js_generic/js_generic.cpp.o
FAILED: CMakeFiles/js_generic.enclave.dir/src/apps/js_generic/js_generic.cpp.o 
/usr/bin/c++  -DCCF_DISABLE_VERBOSE_LOGGING -DCCF_LOGGER_NO_DEPRECATE -DHAVE_ARPA_INET_H=1 -DINSIDE_ENCLAVE -DNGHTTP2_STATICLIB -DOE_BUILD_ENCLAVE -DPLATFORM_SGX -DSGX_ATTESTATION_VERIFICATION -D_LIBCPP_HAS_THREAD_API_PTHREAD -Djs_generic_enclave_EXPORTS -I../include -I../src -I../3rdparty/exported/QCBOR -I../3rdparty/exported/t_cose/inc -I../3rdparty/exported/quickjs -I../3rdparty/exported/nghttp2/includes -isystem ../3rdparty/exported -isystem ../3rdparty/internal -isystem /opt/openenclave/include -isystem /opt/openenclave/include/openenclave/3rdparty/libcxx -isystem /opt/openenclave/include/openenclave/3rdparty/libc -isystem /opt/openenclave/include/openenclave/3rdparty/openssl_3 -isystem /opt/openenclave/include/openenclave/3rdparty -isystem generated -O2 -g -DNDEBUG -fPIC   -fcolor-diagnostics -Wall -Wextra -Werror -Wundef -Wpedantic -Wno-unused -Wno-unused-parameter -Wshadow -mlvi-cfi -nostdinc -nostdinc++ -m64 -fPIE -fstack-protector-strong -fvisibility=hidden -fno-omit-frame-pointer -ffunction-sections -fdata-sections -ftls-model=local-exec -DCONFIG_VERSION=\"2024-01-13\" -DEMSCRIPTEN -DCONFIG_STACK_CHECK -DCONFIG_BIGNUM -std=gnu++2a -MD -MT CMakeFiles/js_generic.enclave.dir/src/apps/js_generic/js_generic.cpp.o -MF CMakeFiles/js_generic.enclave.dir/src/apps/js_generic/js_generic.cpp.o.d -o CMakeFiles/js_generic.enclave.dir/src/apps/js_generic/js_generic.cpp.o -c ../src/apps/js_generic/js_generic.cpp
clang: error: unknown argument: '-mlvi-cfi'
[557/720] Building CXX object CMakeFiles/frontend_test.dir/src/node/rpc/test/frontend_test.cpp.o
ninja: build stopped: subcommand failed.
root@0e857000726d:~/CCF/build# 

To Reproduce

• docker run -it --privileged ghcr.io/microsoft/ccf/ci/default:build-25-07-2024
• Install open-enclave inside docker. Note that I had to --force-overwrite the latest version of open-enclave.
• clone CCF repo from github and checkout ccf-5.0.6
• cmake -GNinja ..
• ninja

Expected behavior
All the builds should be completed successfully.

Environment information

• CCF v5.0.6
• docker environment

-root@0e857000726d:~/CCF/build# c++ --version
clang version 10.0.0-4ubuntu1 
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Additional context
openenclave is present in /opt/openenclave

[achamayou]

@anirbanc3 ghcr.io/microsoft/ccf/ci/default:build-25-07-2024 is for virtual/SNP builds, it confusingly already has an "openenclave" package installed, but that is a cut down variant with only the host side verification library (i.e. the ability to verify SGX quotes, but nothing more). For SGX builds, you want ghcr.io/microsoft/ccf/ci/sgx:build-25-07-2024, which will have the right version of openenclave and of the toolchain installed (with support for the lvi mitigation flag).

[anirbanc3]

Thanks @achamayou for the help. I could build CCF v5.0.6 in the docker ghcr.io/microsoft/ccf/ci/sgx:build-25-07-2024. While running the stress test, some of the test cases failed. Here is the list of the failed tests:

The following tests FAILED:
	 18 - endorsements_test (Failed)
	 48 - recovery_test_cft_api_1 (Failed)
	 49 - recovery_test_suite (Failed)
	 50 - reconfiguration_test_suite (Failed)
	 51 - full_test_suite (Failed)
	 52 - committable_suffix_test (Failed)
	 53 - commit_latency (Failed)
	 54 - js_batched_stress_test (Failed)
	 55 - modules_test (Failed)
	 56 - auth (Failed)
	 57 - launch_host_process_test (Failed)
	 58 - governance_test (Failed)
	 59 - jwt_test (Failed)
	 60 - code_update_test (Failed)
	 61 - e2e_logging_cft (Failed)
	 62 - programmability (Failed)
	 63 - e2e_limits (Failed)
	 64 - e2e_redirects (Failed)
	 65 - e2e_logging_http2 (Failed)
	 66 - membership_api_1 (Failed)
	 67 - connections_cft (Failed)
	 68 - consistency_trace_validation (Failed)
	 69 - fuzz_test (Failed)
	 70 - schema_test_cft (Failed)
	 71 - lts_compatibility (Failed)
	 72 - rotation_test_cft (Failed)
	 73 - reconfiguration_test_cft (Failed)
	 74 - election_test (Failed)
	 75 - pi_ls_sgx (Failed)
	 76 - pi_basic_sgx (Failed)
	 77 - pi_basic_js_sgx (Failed)
	 78 - pi_ls_jwt_sgx (Failed)
	 79 - historical_query_perf_test (Failed)
	 80 - historical_query_cache_test (Failed)

It seems the failed tests are due to error with the Quote_provide_call but I am not able to figure out the exact root cause of the error.

76: 10:26:27.919 | ERROR    | infra.network:log_errors:126 - /root/CCF/build/workspace/pi_basic_sgx_0/out:149: 2024-10-21T10:26:23.042008Z        0   [fail ] ../src/enclave/enclave.h:260         | Error starting node: Failed to get evidence: OE_QUOTE_PROVIDER_CALL_ERROR
76: 10:26:27.919 | ERROR    | infra.network:log_errors:126 - /root/CCF/build/workspace/pi_basic_sgx_0/out:150: 2024-10-21T10:26:23.042369Z        100 [fail ] ../src/host/main.cpp:742             | An error occurred when creating CCF node: InternalError

A detailed log of one of the failed test cases is below:

test 76
      Start 76: pi_basic_sgx

76: Test command: /root/CCF/build/env/bin/python3 "/root/CCF/tests/infra/basicperf.py" "-b" "." "-c" "./submit" "--host-log-level" "info" "--enclave-log-level" "info" "--worker-threads" "0" "--constitution" "/root/CCF/samples/constitutions/default/actions.js" "--constitution" "/root/CCF/samples/constitutions/default/validate.js" "--constitution" "/root/CCF/samples/constitutions/default/resolve.js" "--constitution" "/root/CCF/samples/constitutions/default/apply.js" "--label" "pi_basic_sgx" "--perf-label" "Basic" "--snapshot-tx-interval" "10000" "--package" "samples/apps/basic/libbasic" "--client-def" "1,write,100000,primary" "-e" "release" "-t" "sgx"
76: Environment variables: 
76:  PYTHONPATH=/root/CCF/tests:
76:  TSAN_OPTIONS=suppressions=/root/CCF/tsan_env_suppressions
76: Test timeout computed to be: 10000000
76: 10:26:17.356 | INFO     | __main__:run:224 - Starting nodes on ['local://localhost']
76: 10:26:17.356 | INFO     | infra.network:_setup_common_folder:521 - Creating common folder: /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.356 | INFO     | infra.proc:ccall:11 - rm -rf /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.362 | INFO     | infra.proc:ccall:11 - mkdir -p /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.367 | INFO     | infra.proc:ccall:11 - cp /root/CCF/samples/constitutions/default/actions.js /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.371 | INFO     | infra.proc:ccall:11 - cp /root/CCF/samples/constitutions/default/validate.js /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.376 | INFO     | infra.proc:ccall:11 - cp /root/CCF/samples/constitutions/default/resolve.js /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.380 | INFO     | infra.proc:ccall:11 - cp /root/CCF/samples/constitutions/default/apply.js /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.385 | INFO     | infra.proc:ccall:11 - ln -s /root/CCF/build/keygenerator.sh /root/CCF/build/workspace/pi_basic_sgx_common
76: 10:26:17.389 | INFO     | infra.proc:ccall:11 - /root/CCF/build/keygenerator.sh --name member0 --curve secp384r1 --gen-enc-key [cwd: /root/CCF/build/workspace/pi_basic_sgx_common]
76: 10:26:17.529 | INFO     | infra.member:__init__:351 - Member member0 created: 4fd84e33e4b853a85272935b234e51f4caca0a20ee3f7524807a13a67232b1e1
76: 10:26:17.530 | INFO     | infra.proc:ccall:11 - /root/CCF/build/keygenerator.sh --name member1 --curve secp384r1 --gen-enc-key [cwd: /root/CCF/build/workspace/pi_basic_sgx_common]
76: 10:26:17.740 | INFO     | infra.member:__init__:351 - Member member1 created: d563c16e93704d8f18a9bcaa73dd17ca99c602aa8eaab58deb7fa985f26fdd3e
76: 10:26:17.740 | INFO     | infra.proc:ccall:11 - /root/CCF/build/keygenerator.sh --name member2 --curve secp384r1 --gen-enc-key [cwd: /root/CCF/build/workspace/pi_basic_sgx_common]
76: 10:26:17.783 | INFO     | infra.member:__init__:351 - Member member2 created: b79a59f7ee4f2c0b1f13fb8f715ca5dfc6b875229948b7b2244491be024cf316
76: 10:26:17.783 | DEBUG    | infra.network:_start_all_nodes:449 - Opening CCF service on ['local://localhost']
76: 10:26:17.793 | INFO     | infra.remote:_rc:401 - [127.47.77.77] rm -rf /root/CCF/build/workspace/pi_basic_sgx_0
76: 10:26:17.799 | INFO     | infra.remote:_rc:401 - [127.47.77.77] mkdir -p /root/CCF/build/workspace/pi_basic_sgx_0
76: 10:26:17.804 | INFO     | infra.remote:_rc:401 - [127.47.77.77] ln -s /root/CCF/build/workspace/pi_basic_sgx_common/0.config.json /root/CCF/build/workspace/pi_basic_sgx_0/0.config.json
76: 10:26:17.808 | INFO     | infra.remote:_rc:401 - [127.47.77.77] ln -s /root/CCF/build/cchost /root/CCF/build/workspace/pi_basic_sgx_0/cchost
76: 10:26:17.813 | INFO     | infra.remote:_rc:401 - [127.47.77.77] ln -s /root/CCF/build/samples/apps/basic/libbasic.enclave.so.signed /root/CCF/build/workspace/pi_basic_sgx_0/libbasic.enclave.so.signed
76: 10:26:17.817 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/member0_cert.pem /root/CCF/build/workspace/pi_basic_sgx_0/member0_cert.pem
76: 10:26:17.822 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/member0_enc_pubk.pem /root/CCF/build/workspace/pi_basic_sgx_0/member0_enc_pubk.pem
76: 10:26:17.827 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/member1_cert.pem /root/CCF/build/workspace/pi_basic_sgx_0/member1_cert.pem
76: 10:26:17.832 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/member1_enc_pubk.pem /root/CCF/build/workspace/pi_basic_sgx_0/member1_enc_pubk.pem
76: 10:26:17.837 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/member2_cert.pem /root/CCF/build/workspace/pi_basic_sgx_0/member2_cert.pem
76: 10:26:17.842 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/member2_enc_pubk.pem /root/CCF/build/workspace/pi_basic_sgx_0/member2_enc_pubk.pem
76: 10:26:17.847 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/actions.js /root/CCF/build/workspace/pi_basic_sgx_0/actions.js
76: 10:26:17.852 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/validate.js /root/CCF/build/workspace/pi_basic_sgx_0/validate.js
76: 10:26:17.857 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/resolve.js /root/CCF/build/workspace/pi_basic_sgx_0/resolve.js
76: 10:26:17.862 | INFO     | infra.remote:_rc:401 - [127.47.77.77] cp /root/CCF/build/workspace/pi_basic_sgx_common/apply.js /root/CCF/build/workspace/pi_basic_sgx_0/apply.js
76: 10:26:17.867 | INFO     | infra.remote:start:455 - [127.47.77.77] cd /root/CCF/build/workspace/pi_basic_sgx_0 && ./cchost --config 0.config.json --enclave-log-level info --enclave-file ./libbasic.enclave.so.signed 1> /root/CCF/build/workspace/pi_basic_sgx_0/out 2> /root/CCF/build/workspace/pi_basic_sgx_0/err (env: dict_keys([]))
76: 10:26:27.911 | ERROR    | infra.network:_start_all_nodes:503 - Failed to start node 0
76: Traceback (most recent call last):
76: 
76:   File "/root/CCF/tests/infra/node.py", line 364, in _start
76:     self.remote.get_startup_files(self.common_dir)
76:     │    │      │                 │    └ '/root/CCF/build/workspace/pi_basic_sgx_common'
76:     │    │      │                 └ <infra.node.Node object at 0x7b0dad0dfac0>
76:     │    │      └ <function CCFRemote.get_startup_files at 0x7b0dc5e98820>
76:     │    └ <infra.remote.CCFRemote object at 0x7b0dad0dffd0>
76:     └ <infra.node.Node object at 0x7b0dad0dfac0>
76: 
76:   File "/root/CCF/tests/infra/remote.py", line 933, in get_startup_files
76:     self.remote.get(self.pem, dst_path, timeout=REMOTE_STARTUP_TIMEOUT_S)
76:     │    │      │   │    │    │                 └ 5
76:     │    │      │   │    │    └ '/root/CCF/build/workspace/pi_basic_sgx_common'
76:     │    │      │   │    └ '0.pem'
76:     │    │      │   └ <infra.remote.CCFRemote object at 0x7b0dad0dffd0>
76:     │    │      └ <function LocalRemote.get at 0x7b0dc5e96ca0>
76:     │    └ <infra.remote.LocalRemote object at 0x7b0dad0df940>
76:     └ <infra.remote.CCFRemote object at 0x7b0dad0dffd0>
76: 
76:   File "/root/CCF/tests/infra/remote.py", line 441, in get
76:     raise ValueError(path)
76:                      └ '/root/CCF/build/workspace/pi_basic_sgx_0/0.pem'
76: 
76: ValueError: /root/CCF/build/workspace/pi_basic_sgx_0/0.pem
76: 
76: 
76: The above exception was the direct cause of the following exception:
76: 
76: 
76: Traceback (most recent call last):
76: 
76:   File "/root/CCF/tests/infra/basicperf.py", line 659, in <module>
76:     run(args)
76:     │   └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_de...
76:     └ <function run at 0x7b0dad0d5e50>
76: 
76:   File "/root/CCF/tests/infra/basicperf.py", line 234, in run
76:     network.start_and_open(args)
76:     │       │              └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_de...
76:     │       └ <function Network.start_and_open at 0x7b0dc399d9d0>
76:     └ <infra.network.Network object at 0x7b0dad0dfb80>
76: 
76:   File "/root/CCF/tests/infra/network.py", line 639, in start_and_open
76:     self.start(args, **kwargs)
76:     │    │     │       └ {}
76:     │    │     └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_de...
76:     │    └ <function Network.start at 0x7b0dc399d8b0>
76:     └ <infra.network.Network object at 0x7b0dad0dfb80>
76: 
76:   File "/root/CCF/tests/infra/network.py", line 592, in start
76:     primary = self._start_all_nodes(args, **kwargs)
76:               │    │                │       └ {}
76:               │    │                └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_de...
76:               │    └ <function Network._start_all_nodes at 0x7b0dc399d790>
76:               └ <infra.network.Network object at 0x7b0dad0dfb80>
76: 
76: > File "/root/CCF/tests/infra/network.py", line 462, in _start_all_nodes
76:     node.start(
76:     │    └ <function Node.start at 0x7b0dc39f3dc0>
76:     └ <infra.node.Node object at 0x7b0dad0dfac0>
76: 
76:   File "/root/CCF/tests/infra/node.py", line 221, in start
76:     self._start()
76:     │    └ <function Node._start at 0x7b0dc39fa160>
76:     └ <infra.node.Node object at 0x7b0dad0dfac0>
76: 
76:   File "/root/CCF/tests/infra/node.py", line 368, in _start
76:     raise RuntimeError(
76: 
76: RuntimeError: Error starting node 0
76: 10:26:27.918 | INFO     | infra.network:network:1715 - Stopping network
76: 10:26:27.918 | INFO     | infra.node:stop:469 - Stopping node 0
76: 10:26:27.918 | INFO     | infra.remote:stop:524 - [127.47.77.77] closing
76: 10:26:27.919 | ERROR    | infra.network:log_errors:126 - /root/CCF/build/workspace/pi_basic_sgx_0/out:149: 2024-10-21T10:26:23.042008Z        0   [fail ] ../src/enclave/enclave.h:260         | Error starting node: Failed to get evidence: OE_QUOTE_PROVIDER_CALL_ERROR
76: 10:26:27.919 | ERROR    | infra.network:log_errors:126 - /root/CCF/build/workspace/pi_basic_sgx_0/out:150: 2024-10-21T10:26:23.042369Z        100 [fail ] ../src/host/main.cpp:742             | An error occurred when creating CCF node: InternalError
76: 10:26:27.919 | INFO     | infra.network:log_errors:129 - 2 errors found, printing end of output for context:
76: 10:26:27.919 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.629269Z        100 [info ] ../src/host/main.cpp:444             | Registering RPC interface primary_rpc_interface, on tcp 127.47.77.77:0
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.629289Z        100 [info ] ../src/host/socket.h:49              | TCP RPC Client listening on 127.47.77.77:35097
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.629296Z        100 [info ] ../src/host/main.cpp:458             | Registered RPC interface primary_rpc_interface, on tcp 127.47.77.77:35097
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.629349Z        100 [info ] ../src/host/main.cpp:593             | Startup host time: 2024-10-21 10:26:19.629347
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.629503Z        100 [info ] ../src/host/main.cpp:643             | Creating new node: new network (with 3 initial member(s) and 3 member(s) required for recovery)
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.629508Z        100 [info ] ../src/host/main.cpp:716             | Initialising enclave: enclave_create_node
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.731810Z        0   [info ] ../src/enclave/rpc_sessions.h:244    | Setting max open sessions on interface "primary_rpc_interface" (127.47.77.77:35097) to [1000, 1010] and endorsement authority to Service
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:19.731867Z        0   [info ] ../src/node/node_state.h:1787        | Node TLS connections now accepted
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:23.042008Z        0   [fail ] ../src/enclave/enclave.h:260         | Error starting node: Failed to get evidence: OE_QUOTE_PROVIDER_CALL_ERROR
76: 10:26:27.920 | INFO     | infra.network:log_errors:133 - 2024-10-21T10:26:23.042369Z        100 [fail ] ../src/host/main.cpp:742             | An error occurred when creating CCF node: InternalError
76: 10:26:27.920 | INFO     | infra.network:stop_all_nodes:859 - All nodes stopped
76: Traceback (most recent call last):
76:   File "/root/CCF/tests/infra/node.py", line 364, in _start
76:     self.remote.get_startup_files(self.common_dir)
76:     │                             └ <infra.node.Node object at 0x7b0dad0dfac0>
76:     └ <infra.node.Node object at 0x7b0dad0dfac0>
76:   File "/root/CCF/tests/infra/remote.py", line 933, in get_startup_files
76:     self.remote.get(self.pem, dst_path, timeout=REMOTE_STARTUP_TIMEOUT_S)
76:     │               │         │                 └ 5
76:     │               │         └ '/root/CCF/build/workspace/pi_basic_sgx_common'
76:     │               └ <infra.remote.CCFRemote object at 0x7b0dad0dffd0>
76:     └ <infra.remote.CCFRemote object at 0x7b0dad0dffd0>
76:   File "/root/CCF/tests/infra/remote.py", line 441, in get
76:     raise ValueError(path)
76:                      └ '/root/CCF/build/workspace/pi_basic_sgx_0/0.pem'
76: ValueError: /root/CCF/build/workspace/pi_basic_sgx_0/0.pem
76: 
76: The above exception was the direct cause of the following exception:
76: 
76: Traceback (most recent call last):
76:   File "/root/CCF/tests/infra/basicperf.py", line 659, in <module>
76:     run(args)
76:     │   └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_def=[...
76:     └ <function run at 0x7b0dad0d5e50>
76:   File "/root/CCF/tests/infra/basicperf.py", line 234, in run
76:     network.start_and_open(args)
76:     │                      └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_def=[...
76:     └ <infra.network.Network object at 0x7b0dad0dfb80>
76:   File "/root/CCF/tests/infra/network.py", line 639, in start_and_open
76:     self.start(args, **kwargs)
76:     │          │       └ {}
76:     │          └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_def=[...
76:     └ <infra.network.Network object at 0x7b0dad0dfb80>
76:   File "/root/CCF/tests/infra/network.py", line 592, in start
76:     primary = self._start_all_nodes(args, **kwargs)
76:               │                     │       └ {}
76:               │                     └ Namespace(add_new_node_after_primary_stops=None, binary_dir='/root/CCF/build', ccf_version=None, client='./submit', client_def=[...
76:               └ <infra.network.Network object at 0x7b0dad0dfb80>
76:   File "/root/CCF/tests/infra/network.py", line 462, in _start_all_nodes
76:     node.start(
76:   File "/root/CCF/tests/infra/node.py", line 221, in start
76:     self._start()
76:     └ <infra.node.Node object at 0x7b0dad0dfac0>
76:   File "/root/CCF/tests/infra/node.py", line 368, in _start
76:     raise RuntimeError(
76: RuntimeError: Error starting node 0
76/80 Test #76: pi_basic_sgx .....................***Failed   11.67 sec

[achamayou]

@anirbanc3 is this running in Azure? Is /dev/sgx accessible in the container?

[anirbanc3]

@achamayou I am running on Intel 9th Gen with Ubuntu 22.04. /dev/sgx is present on the host system but its not mounted inside the container, although I am specifying it while launching the container.

sudo docker run --device /dev/sgx -it --privileged ghcr.io/microsoft/ccf/ci/sgx:build-25-07-2024

[achamayou]

@anirbanc3 OE_QUOTE_PROVIDER_CALL_ERROR is what you will generally get if the azure dcap client is missing, or is unable to get an attestation because you are not in Azure (eg. microsoft/Azure-DCAP-Client#174).

If you are not in Azure, you may be able to install and configure the Intel DCAP provider instead, please see #5110 for details.

[anirbanc3]

@achamayou Thanks for the link. I have installed the Intel DCAP (and uninstalled the AZ DCAP) inside the container (as per the instructions provided in https://github.com/openenclave/openenclave/blob/master/docs/GettingStartedDocs/Contributors/NonAccMachineSGXLinuxGettingStarted.md.
But I think the DCAP installation changed the clang and cmake installations that were already present (and working) in the container. Now when I am trying to build ninja again, it gives me the following error:

[1/720] /usr/bin/c++  -DCCF_DISABLE_VERBOSE_LOGGING -DCCF_LOGGER_NO_DEPRECATE -I../include -I../src -isystem ../3rdparty/exported -isystem ../3rdparty/internal -O2 -g -DNDEBUG   -fcolor-diagnostics -std=gnu++2a -MD -MT CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o -MF CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o.d -o CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o -c ../src/crypto/sha256_hash.cpp
FAILED: CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o 
/usr/bin/c++  -DCCF_DISABLE_VERBOSE_LOGGING -DCCF_LOGGER_NO_DEPRECATE -I../include -I../src -isystem ../3rdparty/exported -isystem ../3rdparty/internal -O2 -g -DNDEBUG   -fcolor-diagnostics -std=gnu++2a -MD -MT CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o -MF CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o.d -o CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o -c ../src/crypto/sha256_hash.cpp
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /usr/bin/c++ -DCCF_DISABLE_VERBOSE_LOGGING -DCCF_LOGGER_NO_DEPRECATE -I../include -I../src -isystem ../3rdparty/exported -isystem ../3rdparty/internal -O2 -g -DNDEBUG -fcolor-diagnostics -std=gnu++2a -MD -MT CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o -MF CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o.d -o CMakeFiles/stdcxxccfcrypto.host.dir/src/crypto/sha256_hash.cpp.o -c ../src/crypto/sha256_hash.cpp 
1.	<eof> parser at end of file
2.	../include/ccf/ds/hex.h:52:15: instantiating function definition 'ccf::ds::from_hex<__gnu_cxx::__normal_iterator<unsigned char *, std::vector<unsigned char, std::allocator<unsigned char>>>>'
/lib/x86_64-linux-gnu/libLLVM-11.so.1(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamE+0x1f)[0x7f57334cbd1f]
/lib/x86_64-linux-gnu/libLLVM-11.so.1(_ZN4llvm3sys17RunSignalHandlersEv+0x50)[0x7f57334ca080]
/lib/x86_64-linux-gnu/libLLVM-11.so.1(_ZN4llvm3sys15CleanupOnSignalEm+0xdd)[0x7f57334cb46d]
/lib/x86_64-linux-gnu/libLLVM-11.so.1(+0x9ee570)[0x7f5733413570]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f573a58b420]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang8CastExpr19getSubExprAsWrittenEv+0x8b)[0x7f573854fb2b]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZNK5clang4Expr17isDefaultArgumentEv+0x38)[0x7f5738554208]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x150085e)[0x7f5738ece85e]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x15063e9)[0x7f5738ed43e9]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x1506f1b)[0x7f5738ed4f1b]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x1505f83)[0x7f5738ed3f83]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x14fe082)[0x7f5738ecc082]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x1514ece)[0x7f5738ee2ece]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x1519d99)[0x7f5738ee7d99]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x1514ece)[0x7f5738ee2ece]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang4Sema9SubstStmtEPNS_4StmtERKNS_30MultiLevelTemplateArgumentListE+0x41)[0x7f5738ecc021]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang4Sema29InstantiateFunctionDefinitionENS_14SourceLocationEPNS_12FunctionDeclEbbb+0xdd0)[0x7f5738f058d0]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang4Sema28PerformPendingInstantiationsEb+0x262)[0x7f5738f07da2]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang4Sema33ActOnEndOfTranslationUnitFragmentENS0_14TUFragmentKindE+0x1f6)[0x7f57388ba0c6]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang4Sema25ActOnEndOfTranslationUnitEv+0x136)[0x7f57388ba7e6]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang6Parser17ParseTopLevelDeclERNS_9OpaquePtrINS_12DeclGroupRefEEEb+0x411)[0x7f57383758a1]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang8ParseASTERNS_4SemaEbb+0x1fd)[0x7f57382d83dd]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang14FrontendAction7ExecuteEv+0x48)[0x7f57398a42f8]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang16CompilerInstance13ExecuteActionERNS_14FrontendActionE+0x861)[0x7f573985a5f1]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang25ExecuteCompilerInvocationEPNS_16CompilerInstanceE+0x5e0)[0x7f5739909c80]
/usr/bin/c++(_Z8cc1_mainN4llvm8ArrayRefIPKcEES2_Pv+0x94f)[0x4125ff]
/usr/bin/c++[0x410a3e]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(+0x1bbf4b2)[0x7f573958d4b2]
/lib/x86_64-linux-gnu/libLLVM-11.so.1(_ZN4llvm20CrashRecoveryContext9RunSafelyENS_12function_refIFvvEEE+0xdd)[0x7f573341337d]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZNK5clang6driver10CC1Command7ExecuteEN4llvm8ArrayRefINS2_8OptionalINS2_9StringRefEEEEEPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPb+0x13f)[0x7f573958c9df]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZNK5clang6driver11Compilation14ExecuteCommandERKNS0_7CommandERPS3_+0x2df)[0x7f57395647bf]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZNK5clang6driver11Compilation11ExecuteJobsERKNS0_7JobListERN4llvm15SmallVectorImplISt4pairIiPKNS0_7CommandEEEE+0x77)[0x7f5739564967]
/lib/x86_64-linux-gnu/libclang-cpp.so.11(_ZN5clang6driver6Driver18ExecuteCompilationERNS0_11CompilationERN4llvm15SmallVectorImplISt4pairIiPKNS0_7CommandEEEE+0xdc)[0x7f57395784ec]
/usr/bin/c++(main+0x2533)[0x4104e3]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f5732473083]
/usr/bin/c++(_start+0x2e)[0x40dcde]
clang: error: clang frontend command failed due to signal (use -v to see invocation)
Ubuntu clang version 11.1.0-++20211011094159+1fdec59bffc1-1~exp1~20211011214622.5
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
clang: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: /tmp/sha256_hash-2e33b6.cpp
clang: note: diagnostic msg: /tmp/sha256_hash-2e33b6.sh
clang: note: diagnostic msg: 

********************

[anirbanc3]

@achamayou The installation of openenclave changes the cmake and clang installations of the container. Below is a snap of the command openenclave/ansible-playbook scripts/ansible/oe-contributors-setup.yml

TASK [linux/openenclave : Create symbolic links to clang (previous version)] ******************************************************************************************************************************************************************************************************
skipping: [localhost] => (item={'path': '', 'dest': 'clang-10_symlink'}) 
skipping: [localhost] => (item={'path': '', 'dest': 'clang++-10_symlink'}) 
skipping: [localhost]

TASK [linux/openenclave : Create symbolic links to clang (current version)] *******************************************************************************************************************************************************************************************************
ok: [localhost] => (item={'path': '/usr/bin/clang-11', 'dest': 'clang-11_symlink'})
ok: [localhost] => (item={'path': '/usr/bin/clang++-11', 'dest': 'clang++-11_symlink'})

TASK [linux/openenclave : Create gcc wrapper] *************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [linux/openenclave : Create g++ wrapper] *************************************************************************************************************************************************************************************************************************************
changed: [localhost]

TASK [linux/openenclave : Create symbolic links to gcc] ***************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item={'path': '/usr/bin/gcc', 'dest': 'gcc_symlink'})
ok: [localhost] => (item={'path': '/usr/bin/g++', 'dest': 'g++_symlink'})

TASK [linux/openenclave : Install CMake 3.13.1] ***********************************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [linux/openenclave : Create CMake symbolic links] ****************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=ccmake)
ok: [localhost] => (item=cmake)
ok: [localhost] => (item=cpack)
ok: [localhost] => (item=ctest)