Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

launch_local_testnet.sh: Lighthouse VC nodes #3477

Merged
merged 2 commits into from
Mar 11, 2022
Merged

Conversation

stefantalpalaru
Copy link
Contributor

@stefantalpalaru stefantalpalaru commented Mar 9, 2022

Using LH VC nodes:
./scripts/launch_local_testnet.sh --nodes 4 --stop-at-epoch 5 --disable-htop --lighthouse-vc-nodes 2 -- --verify-finalization --discv5:no

For the next runs, do yourself a favour and add --reuse-binaries before --. They'll be built once more, because they've just been deleted.

Problems so far:

  • missing REST API call (non-fatal): {"lvl":"DBG","ts":"2022-03-09 11:52:54.994+01:00","msg":"Request is not part of API","topics":"beacnde","peer":"127.0.0.1:57826","meth":"GET","uri":"/lighthouse/staking"}
  • a more serious looking one, on the LH VC side: Mar 09 12:06:21.019 CRIT Error during attestation routine slot: 65, committee_index: 0, error: "All endpoints failed http://127.0.0.1:7500/ => Unavailable(Offline), http://127.0.0.1:7500/ => Unavailable(Offline)", service: attestation (BN was offline due to the assertion failure)
  • BN assertion failure:
{"lvl":"DBG","ts":"2022-03-09 12:06:17.328+01:00","msg":"batch crypto - failure, falling back","topics":"gossip_checks","items":1}
../csu/libc-start.c(409)
../sysdeps/nptl/libc_start_call_main.h(58)
/mnt/sde1/storage/nimbus-eth2/vendor/nim-json-rpc/json_rpc/client.nim(381) main
/mnt/sde1/storage/nimbus-eth2/vendor/nim-json-rpc/json_rpc/client.nim(374) NimMain
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1930) main
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1799) handleStartUpCmd
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1644) doRunBeaconNode
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1456) start
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1400) run
/mnt/sde1/storage/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(279) poll
/mnt/sde1/storage/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(365) futureContinue
/mnt/sde1/storage/nimbus-eth2/beacon_chain/gossip_processing/gossip_validation.nim(170) validateAttestation
/mnt/sde1/storage/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/assertions.nim(22) raiseAssert
/mnt/sde1/storage/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(49) sysFatal
[[reraised from:
../csu/libc-start.c(409)
../sysdeps/nptl/libc_start_call_main.h(58)
/mnt/sde1/storage/nimbus-eth2/vendor/nim-json-rpc/json_rpc/client.nim(381) main
/mnt/sde1/storage/nimbus-eth2/vendor/nim-json-rpc/json_rpc/client.nim(374) NimMain
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1930) main
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1799) handleStartUpCmd
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1644) doRunBeaconNode
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1456) start
/mnt/sde1/storage/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1400) run
/mnt/sde1/storage/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(279) poll
/mnt/sde1/storage/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(387) futureContinue
]]
Error: unhandled exception: Attestation: invalid signature [AssertionError]

@stefantalpalaru
Copy link
Contributor Author

stefantalpalaru commented Mar 9, 2022

Without --verify-finalization, the BN says this:

{"lvl":"DBG","ts":"2022-03-09 14:21:24.234+01:00","msg":"Dropping aggregate","error":{"Field0":1,"Field1":"Aggregate: invalid aggregator signature"},"aggregate":{"aggregation_bits":"0x12","data":{"slot":76,"index":0,"beacon_block_root":"9c20f709","source":"0:00000000","target":"2:b441d24a"},"signature":"967763b9"},"wallSlot":76,"aggregator_index":91,"signature":"ad332aff","selection_proof":"94bbfbb5"}
{"lvl":"DBG","ts":"2022-03-09 14:21:56.350+01:00","msg":"Dropping attestation","validationError":{"Field0":1,"Field1":"Attestation: invalid signature"},"wallSlot":79,"subnet_id":15,"attestation":{"aggregation_bits":"0x14","data":{"slot":79,"index":0,"beacon_block_root":"f392cf7e","source":"0:00000000","target":"2:b441d24a"},"signature":"886acf75"}}
{"lvl":"NOT","ts":"2022-03-09 14:21:56.350+01:00","msg":"Produced attestation failed validation","topics":"beacval","attestation":{"aggregation_bits":"0x14","data":{"slot":79,"index":0,"beacon_block_root":"f392cf7e","source":"0:00000000","target":"2:b441d24a"},"signature":"886acf75"},"error":{"Field0":1,"Field1":"Attestation: invalid signature"}}
{"lvl":"DBG","ts":"2022-03-09 14:21:56.351+01:00","msg":"Received error response from handler","topics":"beacnde","status":400,"meth":"POST","peer":"127.0.0.1:59338","uri":"/eth/v1/beacon/pool/attestations","error_status":"400 Bad Request","error_content_type":"application/json","error_message":"{\"code\":\"400\",\"message\":\"Some errors happened while validating attestation(s)\",\"failures\":[{\"index\":\"0\",\"message\":\"Attestation: invalid signature\"}]}"}
{"lvl":"DBG","ts":"2022-03-09 14:22:08.234+01:00","msg":"Dropping attestation","validationError":{"Field0":1,"Field1":"Attestation: invalid signature"},"wallSlot":80,"subnet_id":16,"attestation":{"aggregation_bits":"0x14","data":{"slot":80,"index":0,"beacon_block_root":"5aa47e2a","source":"0:00000000","target":"2:b441d24a"},"signature":"aea56861"}}
{"lvl":"NOT","ts":"2022-03-09 14:22:08.234+01:00","msg":"Produced attestation failed validation","topics":"beacval","attestation":{"aggregation_bits":"0x14","data":{"slot":80,"index":0,"beacon_block_root":"5aa47e2a","source":"0:00000000","target":"2:b441d24a"},"signature":"aea56861"},"error":{"Field0":1,"Field1":"Attestation: invalid signature"}}
{"lvl":"DBG","ts":"2022-03-09 14:22:08.234+01:00","msg":"Received error response from handler","topics":"beacnde","status":400,"meth":"POST","peer":"127.0.0.1:59340","uri":"/eth/v1/beacon/pool/attestations","error_status":"400 Bad Request","error_content_type":"application/json","error_message":"{\"code\":\"400\",\"message\":\"Some errors happened while validating attestation(s)\",\"failures\":[{\"index\":\"0\",\"message\":\"Attestation: invalid signature\"}]}"}

And the LH VC says this:

{"msg":"Error whilst producing block","level":"CRIT","ts":"2022-03-09T14:19:28.102102578+01:00","service":"block","message":"All endpoints failed http://127.0.0.1:7500/ => RequestFailed(\"Error from beacon node when producing block: Reqwest(reqwest::Error { kind: Decode, source: Error(\\\"data did not match any variant of untagged enum BeaconBlock\\\", line: 1, column: 4120) })\")"}

@github-actions
Copy link

github-actions bot commented Mar 9, 2022

Unit Test Results

     12 files  ±0     829 suites  ±0   25m 3s ⏱️ - 4m 11s
1 674 tests ±0  1 626 ✔️ ±0    48 💤 ±0  0 ±0 
9 775 runs  ±0  9 663 ✔️ ±0  112 💤 ±0  0 ±0 

Results for commit 18f2599. ± Comparison against base commit 266fd98.

♻️ This comment has been updated with latest results.

@arnetheduck
Copy link
Member

Without --verify-finalization, the BN says this:

did this get investigated / solved? what was the root cause?

@stefantalpalaru stefantalpalaru merged commit 857a71b into unstable Mar 11, 2022
@stefantalpalaru stefantalpalaru deleted the lh_vc branch March 11, 2022 12:44
@stefantalpalaru
Copy link
Contributor Author

It can be investigated and solved more easily after being merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants