Spamming Westend causes grandpa-voter to fail

[tugytur]

In preparation for the spamming the Relaychains we were testing it on Westend.

After couple seconds of running the blocktime went >1 minute.

I don't have access to the validator logs/metrics, but thankfully @lovelaced checked.

This error showed up.

@lovelaced and I both suspect it could be caused by the low compute specs of the Westend valiators.

Would appreciate feedback if we should continue on Kusama or if Parity wants to first check some things on Westend.
For Kusama, we would first do a 1-3 minute run before doing the full 30 minutes couple days later.

If any more logs are required please reach out to @lovelaced and just let me know if you want me to run it on Westend again.

[lovelaced]

I realized the first graph is for Westend AH (my fault) but the story is the same on the validators.

[bkchr]

We need more logs. @lovelaced please enable grandpa=trace.

After that is done we need the script again @tugytur

[BulatSaif]

We need more logs. @lovelaced please enable grandpa=trace.

After that is done we need the script again @tugytur

grandpa=trace added to westend-validators: https://grafana.teleport.parity.io/goto/TXBj7i7HR?orgId=1

[lexnv]

The gossip engine exists when either the notification event stream closes or the sync event stream closes:

polkadot-sdk/substrate/client/network-gossip/src/bridge.rs

Lines 247 to 251 in 04e2bd3

	// The network event stream closed. Do the same for [`GossipValidator`].
	Poll::Ready(None) => {
	self.is_terminated = true;
	return Poll::Ready(())
	},

polkadot-sdk/substrate/client/network-gossip/src/bridge.rs

Lines 262 to 266 in 04e2bd3

	// The sync event stream closed. Do the same for [`GossipValidator`].
	Poll::Ready(None) => {
	self.is_terminated = true;
	return Poll::Ready(())
	},

Have detected a poll mismatch implementation, although I don't expect that to be the root cause of this issue:

• network-gossip: Ensure sync event is processed on unknown peer roles  #6553

We'll probably need to collect more logs.

@tugytur Would it be possible to make the script public? Would like to run this in versi-net to check litep2p as well 🙏

[bkchr]

@lexnv https://github.com/amforc/spammening but it was hard to get it running. We probably need some coordinated effort together with @tugytur to run it again on westend.