Thread 'tokio-runtime-worker' panicked at 'Critical database error // Too many open files -- 2409-2 upgrade

[infrachris]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

While starting up for the first time after upgrading to 2409-2, I received the following error:
Thread 'tokio-runtime-worker' panicked at 'Critical database error: Custom { kind: Other, error: Error { message: "IO error: While open a file for random read: /home/polkadot/.local/share/polkadot/chains/polkadot/db/full/4304564.sst: Too many open files" } }', /builds/substrate/primitives/database/src/kvdb.rs:29 This is a bug. Please report it at: https://github.com/paritytech/polkadot-sdk/issues/new polkadot.service: Main process exited, code=exited, status=1/FAILURE polkadot.service: Failed with result 'exit-code'. polkadot.service: Consumed 1min 23.427s CPU time.

The node failed to stay running upon the start where this error was received and went into a failed/stopped state. Node stayed running after a subsequent start.

I have ulimits set to 10,000, and wait 120seconds between stopping and starting.

Steps to reproduce

This occurred while upgrading just one of my nodes, but it was the only polkadot node I have (RocksDB). I will work on posting more detail along with some surrounding error messages shortly.

[TheCrazyStaker]

I think this error tokio-runtime-worker' panicked has nothing to do with upgrading to 2409-2. I got the same error 1 day before 2409-2 was released.

[bkchr]

I have ulimits set to 10,000, and wait 120seconds between stopping and starting.

If you increase it to 20000, does it fix it?

[infrachris]

I have ulimits set to 10,000, and wait 120seconds between stopping and starting.

If you increase it to 20000, does it fix it?

It hasn't been very repeatable as just attempting to start the service again did not show the error. I'll try 20000 if it happens again. I'm also going to dig much more if there's a hidden ulimit or something going on, and update or close this. Thank you

[bkchr]

The node automatically raises the limit to maximum that is possible without root rights. It also prints a warning if this target limit is below 10_000. However, I think these 10_000 are probably more like a random number that we maybe need to increase now.

[infrachris]

I've changed ulimit to 1048576, I'll update and re-open if this comes up again.