Logs not printed when there is an error during shutdown

[skunert]

Issue Description

In some runs the links to the logs in loki are not printed.
See here: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/4513620

It looks like this is due to some shutdown error (one pod could not be found).
This makes debugging of these jobs impossible, logs are also not under the artifacts folder.

Steps to reproduce the issue

Describe the results you received

Logs not available

Describe the results you expected

Logs available

Zombienet version

1.3.83

Provider

Kubernetes

Provider version

CI

Upstream Latest Release

Yes

Additional environment details

CI

Additional information

Only occasionally

Screenshots

No response

[pepoviola]

Hi @skunert, thanks for reporting. Yes, in this case there was an unhandled exception and the process crash. I will fix asap, as workaround until is fixed you can see the logs in loki by namespace
https://grafana.teleport.parity.io/goto/434r-pNSg?orgId=1

Thanks!!

[mrcnski]

@pepoviola I might be getting something similar. Zombienet never errors for me, even when the test is clearly wrong.

For example, I change some value in 0001-parachains-pvf.zndsl, let's say I set alice: reports node_roles is 3. AFAICT zombienet never exits, it just keeps running as if everything is fine. I tried changing various lines in various tests.

Reopening since I think this ticket is related.

 $ zombienet version
1.3.86

[pepoviola]

Hi @mrcnski, thanks for reopen. Are you trying locally with the native provider or those changes are in CI? I will try to reproduce and fix asap.

Thanks!

[mrcnski]

Thanks @pepoviola, appreciate it! I'm trying locally with native. I wonder, is there a way to get logs out of zombienet to see what it's doing?

[mrcnski]

@pepoviola Could this be the issue? (changed from Promise.all in tag 1.3.85):

zombienet/javascript/packages/orchestrator/src/network.ts

Line 174 in 880a556

await Promise.allSettled(dumpsPromises);

I checked out 1.3.84 and I see errors now, but unfortunately I get stuck on the dreaded Error fetching metrics from loop.

[pepoviola]

Hey @mrcnski, yes I made that change to prevent the rejection off all dumps if one fails. I would check why doesn't resolve as expected or handle the rejection internally in the dump fn to prevent the issue using all

Error fetching metrics. This error is caused mostly when the node didn't start as expected. Can you check the logs of the node? We just make a get to the prometheus endpoint to check the readyness.

Thanks!!

[mrcnski]

Thanks @pepoviola! Looks like the metrics error was caused by the nodes missing the insecure flag I added - oops!

I'm back to not seeing any errors anymore. I mean, I added a ridiculous line like alice: parachain 2000 block height is at least 1000 within 2 seconds, shouldn't zombienet exit with an error? I can't seem to make it fail. I'm still on version 1.3.84.

[pepoviola]

Hey @mrcnski, great that now the metrics error is gone. Zombienet will continue working until the end of the last defined assertion and produce a report in the console. Are you having more assertion beside tha one? Can you send me the config and zndsl to reproduce?

Thanks!!

[mrcnski]

@pepoviola For a really simple case I've changed 0001-parachains-smoke-test.zndsl to something crazy like

Description: Smoke Test
Network: ./0001-parachains-smoke-test.toml
Creds: config

alice: parachain 100 is registered within 2 seconds
alice: parachain 100 block height is at least 1000000 within 3 seconds

and I run with

zombienet --provider=native spawn zombienet_tests/smoke/0001-parachains-smoke-test.toml 

Zombienet doesn't fail on this. I checked the logs of Alice and she's still alive doing stuff.

[pepoviola]

Hi @mrcnski, I can reproduce locally. In my end changing the value produce this output

Also, you are using the spawn command in your comment. That command only spawn the network and don't make any assertion. To run the test you should run something like

zombienet --provider=native test zombienet_tests/smoke/0001-parachains-smoke-test.zndsl

Thanks!!

[mrcnski]

Let's pretend this never happened.