whitelisting milestone deferred err="chain out of sync"

[srvint]

Our support team has aggregated some common issues and their solutions from past which are faced while running or interacting with a bor client. In order to prevent redundant efforts, we would encourage you to have a look at the FAQ's section of our documentation mentioning the same, before filing an issue here. In case of additional support, you can also join our discord server

System information

Bor client version: [e.g. v0.2.16] 1.4.1

Heimdall client version: [e.g. v0.2.10] 1.0.3

OS & Version: Linux

Environment: Polygon Mainnet

Type of node: Archive

Additional Information:

Overview of the problem

Multiple nodes, run fine but eventually they receive this message and chain stops syncing. Sometimes restarting the chain fixes the issue, sometimes it does not.
Whitelisting checkpoint deferred err="chain out of sync"
Whitelisting milestone deferred err="chain out of sync"

Reproduction Steps

Start bor after heimdall is caught up:
./bor server --config ./config.toml

Logs / Traces / Output / Error Messages

INFO [10-10|10:29:33.272] Fetching state updates from Heimdall fromID=2,968,492 to=2024-10-10T08:40:33-05:00
INFO [10-10|10:29:33.272] Fetching state sync events queryParams="from-id=2968492&to-time=1728567633&limit=50"
INFO [10-10|10:29:33.273] Fetching state updates from Heimdall fromID=2,968,492 to=2024-10-10T08:40:33-05:00
INFO [10-10|10:29:33.273] Fetching state sync events queryParams="from-id=2968492&to-time=1728567633&limit=50"
INFO [10-10|10:29:33.278] StateSyncData gas=0 number=62,867,440 lastStateID=2,968,491 "total records"=0 "fetch time"=12 "process time"=0
INFO [10-10|10:29:33.278] StateSyncData gas=0 number=62,867,440 lastStateID=2,968,491 "total records"=0 "fetch time"=12 "process time"=0
INFO [10-10|10:29:33.851] Whitelisting milestone deferred err="chain out of sync"
INFO [10-10|10:29:35.491] Coinbase is in MVReadMap address=0x1EFEcb61A2f80Aa34d3b9218B564a64D05946290
INFO [10-10|10:29:35.518] Coinbase is in MVReadMap address=0x1EFEcb61A2f80Aa34d3b9218B564a64D05946290
INFO [10-10|10:29:35.947] Imported new chain segment number=62,867,452 hash=cea300..8b3e57 blocks=43 txs=3128 mgas=555.902 elapsed=8.079s mgasps=68.807 age=1h46m30s snapdiffs=7.58MiB triedirty=1022.63MiB
WARN [10-10|10:29:36.286] Synchronisation failed, dropping peer peer=46e086a9e8ce21511ad18cce9e5c2cbc34c6ffb07bb7f38b02183720149787c9 err="retrieved hash chain is invalid: out of gas"
WARN [10-10|10:29:36.286] Synchronisation failed, retrying peer=46e086a9e8ce21511ad18cce9e5c2cbc34c6ffb07bb7f38b02183720149787c9 err="peer is unknown or unhealthy"
WARN [10-10|10:29:36.286] Synchronisation failed, retrying peer=46e086a9e8ce21511ad18cce9e5c2cbc34c6ffb07bb7f38b02183720149787c9 err="peer is unknown or unhealthy"
INFO [10-10|10:29:36.308] Commit Interrupt. Pre-committing the current block block=62,867,455
WARN [10-10|10:29:36.327] Tx Level Interrupt hash=000000..000000
INFO [10-10|10:29:45.850] Whitelisting milestone deferred err="chain out of sync"
INFO [10-10|10:29:49.853] Whitelisting checkpoint deferred err="chain out of sync"
INFO [10-10|10:29:57.851] Whitelisting milestone deferred err="chain out of sync"
INFO [10-10|10:30:09.851] Whitelisting milestone deferred err="chain out of sync"
INFO [10-10|10:30:21.850] Whitelisting milestone deferred err="chain out of sync"

Additional Information

maxpeers = 400

bootnodes = [
"enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303",
"enode://bdcd4786a616a853b8a041f53496d853c68d99d54ff305615cd91c03cd56895e0a7f6e9f35dbf89131044e2114a9a782b792b5661e3aff07faf125a98606a071@43.200.206.40:30303",
"enode://209aaf7ed549cf4a5700fd833da25413f80a1248bd3aa7fe2a87203e3f7b236dd729579e5c8df61c97bf508281bae4969d6de76a7393bcbd04a0af70270333b3@54.216.248.9:30303",
"enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303",
"enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"
]
static-nodes = [
"enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303",
"enode://bdcd4786a616a853b8a041f53496d853c68d99d54ff305615cd91c03cd56895e0a7f6e9f35dbf89131044e2114a9a782b792b5661e3aff07faf125a98606a071@43.200.206.40:30303",
"enode://209aaf7ed549cf4a5700fd833da25413f80a1248bd3aa7fe2a87203e3f7b236dd729579e5c8df61c97bf508281bae4969d6de76a7393bcbd04a0af70270333b3@54.216.248.9:30303",
"enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303",
"enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"
]
trusted-nodes = [
"enode://429bb1c348c769218ecf2b400dc16b86045f3eec89780a8d8d73913f6c9dc184a8d3d3bfcb7197092065c2ebbad4fd177aa8d9053f5051e5db518f541144268a@46.4.95.132:30303",
"enode://bdcd4786a616a853b8a041f53496d853c68d99d54ff305615cd91c03cd56895e0a7f6e9f35dbf89131044e2114a9a782b792b5661e3aff07faf125a98606a071@43.200.206.40:30303",
"enode://209aaf7ed549cf4a5700fd833da25413f80a1248bd3aa7fe2a87203e3f7b236dd729579e5c8df61c97bf508281bae4969d6de76a7393bcbd04a0af70270333b3@54.216.248.9:30303",
"enode://b8f1cc9c5d4403703fbf377116469667d2b1823c0daf16b7250aa576bacf399e42c3930ccfcb02c5df6879565a2b8931335565f0e8d3f8e72385ecf4a4bf160a@3.36.224.80:30303",
"enode://8729e0c825f3d9cad382555f3e46dcff21af323e89025a0e6312df541f4a9e73abfa562d64906f5e59c51fe6f0501b3e61b07979606c56329c020ed739910759@54.194.245.5:30303"
]

[andreyvg]

Did you check the DNS discovery?
https://forum.polygon.technology/t/introducing-our-new-dns-discovery-for-polygon-pos-faster-smarter-more-connected/19871

[srvint]

This solved our problem. Thanks!

[yuanwj210]

您检查过 DNS 发现吗？ https://forum.polygon.technology/t/introducing-our-new-dns-discovery-for-polygon-pos-faster-smarter-more-connected/19871

After adding DNS, I still can't keep up, and I get stuck every time I keep up
Whitelisting checkpoint deferred err="chain out of sync"