Simulating large scenarios / parallel execution

[exocet-dev]

Hi all,

I'm working on a large scenario involving a one-year timeframe with a high number of flights per day. I'm seeking efficient strategies to handle such large-scale simulations with respect to computation time, as I have access to parallel computing resources.

From my understanding, there are no built-in parallel execution methods in pycontrails. I've attempted to brute-force parallelize cocip.eval using tools like joblib and multiprocessing, but this approach significantly slows down the calculations. I suspect the issue is memory-bound, particularly due to accessing the weather data variables.

One potential workaround I've considered is splitting the data into daily simulations and running each day in parallel. However, I would appreciate any additional experiences or suggestions on handling such large scenarios more efficiently.

Thank you!

[mlshapiro]

We do have support to run multiple flights in parallel with the same set of meteorogoly (see example Run CoCiP on multiple flights). This is much more efficiency than running one flight at a time.

But for simulations where the meteorology won't fit into memory, there aren't built-in parallel execution managers using multiprocessing, joblib, etc. We've explored the area extensively for running the simuliations behind the Teoh 2024 Global contrail simulation, but the patterns aren't general enough to include in the library.

Your suggested workaround has been our general approach to the probelm as well - For the very large simulations, we'll find a chunk of meteorology that is small enough to fit in memory (generally 1 - 2 days like you suggest) and run all the flights in the domain. Then we'll parallelize these jobs in separate processes (or VMs).

@zebengberg, @roger-teoh or @thabbott feel free to chime in here if have other thoughts or examples

[thabbott]

I agree with @mlshapiro's recommendations.

One additional note: I've never been able to figure out a way to share resources between processes in a way that allowed me to use multiprocessing to parallelize Cocip.eval alone. If you go the multiprocessing route, I'd recommend structuring the code so that each process creates its own MetDataset(s) and Cocip instance--i.e., so that they share as few resources as possible.

[exocet-dev]

Thanks for your swift response!
Just to understand it correctly: when I run Cocip.eval over a list of flights (that's what I typically do), this list is being processed in parallel? Would that simply use all the available cores on the machine or is there a way of specifing/tracking the used ressources?

[thabbott]

I guess Marc and I were using "parallel" to mean different things--sorry for the confusion!

If you run Cocip.eval over a list of flights the flights will be processed simultaneously (meaning all the contrails produced by all the flights will be simulated in a single timestepping loop), but still only in a single thread. If you want to run on multiple cores in parallel you'll have to split flights up yourself (e.g., by day, like you described) and start multiple simulations in different processes.