v1.4.0: submitPoolAttestations errors

[twoeths]

Describe the bug

Sometimes we got this error since v1.4.0

1. ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT (rarely)

Jan-31 16:42:05.628[rest]            ^[[31merror^[[39m: Req req-jcs submitPoolAttestations error  Multiple errors on submitPoolAttestations
ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT
ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT
ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT
Error: Multiple errors on submitPoolAttestations
ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT
ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT
ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:75:15)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

2. PublishError.Duplicate (more frequent)

Jan-31 20:19:41.932[rest]            ^[[31merror^[[39m: Req req-mbm submitPoolAttestations error  Multiple errors on submitPoolAttestations
PublishError.Duplicate
PublishError.Duplicate
PublishError.Duplicate
Error: Multiple errors on submitPoolAttestations
PublishError.Duplicate
PublishError.Duplicate
PublishError.Duplicate
    at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:75:15)

Expected behavior

No error

Desktop (please complete the following information):

• Version: v1.4.0
• Network: mainnet

[avenbreaks]

same issue maybe

Feb-03 11:57:19.166[api]             �[31merror�[39m: Error on publishContributionAndProofs [0] slot=32, subcommitteeIndex=3 PublishError.Duplicate
Error: PublishError.Duplicate
    at Eth2Gossipsub.publish (file:///mnt/block-volume/lodestar/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1950:13)
    at Eth2Gossipsub.publishObject (file:///mnt/block-volume/lodestar/packages/beacon-node/src/network/gossip/gossipsub.ts:171:20)
    at Eth2Gossipsub.publishContributionAndProof (file:///mnt/block-volume/lodestar/packages/beacon-node/src/network/gossip/gossipsub.ts:263:5)
    at file:///mnt/block-volume/lodestar/packages/beacon-node/src/api/impl/validator/index.ts:566:13
    at async Promise.all (index 0)
    at Object.publishContributionAndProofs (file:///mnt/block-volume/lodestar/packages/beacon-node/src/api/impl/validator/index.ts:556:7)
    at Object.handler (file:///mnt/block-volume/lodestar/packages/api/src/utils/server/genericJsonServer.ts:41:23)
Feb-03 11:57:19.168[rest]            �[31merror�[39m: Req req-89 publishContributionAndProofs error  Multiple errors on publishContributionAndProofs
PublishError.Duplicate
PublishError.Duplicate
Error: Multiple errors on publishContributionAndProofs
PublishError.Duplicate
PublishError.Duplicate

[twoeths]

1. ATTESTATION_ERROR_UNKNOWN_OR_PREFINALIZED_BEACON_BLOCK_ROOT

This is because validator requests attestation data from one node (block at slot comes within 4s) and publish signed attestation to another node (block at slot hasn't come yet)

found this in the log

grep -e "Requesting fallback URL routeId=produceAttestationData" -rn validator*.log | wc -l
296

in this case, the submitPoolAttestations api from validator should not be rejected because the request should be successful for the original node

Update: confirm that this is a success submitPoolAttestations call from validator side, the log in validator is just debug log, attestation is not missed. At the beacon node side it's still an error since the beacon node may not know the attestation head block

[twoeths]

2. PublishError.Duplicate

This is because the way we implement HttpClient is to issue multiple requests in parallel, sometimes attestations reach one node then gossip to another node, and the api at the other node would fail with Duplicate error, but overall the api is a success at validator side

perhaps we could ignore duplicate error for the publish api once this is implemented (add a flag to beacon node if needed) ChainSafe/js-libp2p-gossipsub#402

[twoeths]

@avenbreaks could you confirm if you configured fallback urls at your validator side? the beaconNodes option is something like beaconNodes: "http://ip_1:9596,http://ip_2:9596"

[maschad]

2. PublishError.Duplicate

This is because the way we implement HttpClient is to issue multiple requests in parallel, sometimes attestations reach one node then gossip to another node, and the api at the other node would fail with Duplicate error, but overall the api is a success at validator side

perhaps we could ignore duplicate error for the publish api once this is implemented (add a flag to beacon node if needed) ChainSafe/js-libp2p-gossipsub#402

Seems like a similar issue to #4135 perhaps we could allow publishing the same message for multiple topics?

[twoeths]

Seems like a similar issue to #4135 perhaps we could allow publishing a the same message for multiple topics?

you're right, right now we just call publish() multiple times for different subnets. We need to ignore the duplicate check in gossipsub publish() api anyway