Use ClimaUtilities.OutputPathGenerator to organize output directories

[Sbozzolo]

When a simulation is run with the driver, remove existing output. This is helpful when dealing with diagnostics, given that NetCDF output is appended.

I personally think that this is a dangerous feature to have, and we should probably think of a safer solution for the long term.

See comment below about ClimaUtilities

[szy21]

@trontrytel What do you think?

[trontrytel]

Would it be possible to automatically name the output directory with the current date and time? This will prevent us from trying to append to files that already exist. And would not require automatically delete everything?

If thats not a good idea then I'm fine with it. I keep changing folder names manually anyway to deal with the simulation trying to append

[Sbozzolo]

I changed it to move it to output_dir_YYYYMMDD_HHMM (with the current time). The latest simulation will always be in the same path, but the previous one will be "archived". The drawback is that it is up to you to keep your folder clean.

I think this is a much better solution, but you will have to do spring-cleaning every so often.

[Sbozzolo]

Do we want to merge this?

[szy21]

Sounds good to me, thanks! (In the long-term I have some other ideas about output file names, we can talk about that later)

[charleskawczynski]

This will break our reproducibiity tests. Why is this needed exactly?

[Sbozzolo]

This will break our reproducibiity tests. Why is this needed exactly?

The issue that this PR is solving is this:

If you run the driver multiple times with the same configuration file, the same output folder is used. Data from different simulations is mixed, and NetCDF files from different simulations get appended to each others. For interactive work, this is really annoying.

With this PR, when you run the driver and an output folder already exists, then the preexisting folder gets moved to a different location and a new one is created.

I don't think that this would break the reproducibility tests: this PR changes the behavior of the driver exclusively when there is a preexisting output folder where you are running a new simulations.

I also don't think that this logic should be in /src. I think that the most convervative behvior is to error out if the output folder already exists: touching existing users' files is never a good idea.

[charleskawczynski]

I don't think that this would break the reproducibility tests: this PR changes the behavior of the driver exclusively when there is a preexisting output folder where you are running a new simulations.

Ok, this makes sense, and I agree that this shouldn't break reproducibility tests (right now). This workflow really won't work well for reproducibility tests for restarted simulations (#2604), though.

If you run the driver multiple times with the same configuration file, the same output folder is used. Data from different simulations is mixed, and NetCDF files from different simulations get appended to each others. For interactive work, this is really annoying.

Why is data from different simulations mixed? And why are NC files from different simulations appended? Can't we instead change that logic? This workflow is equally annoying to me: running a simulation 10x interactively will generate 10x the amount of data compared to just overwriting old files. Isn't that what the restart flag is for?

[Sbozzolo]

I don't think that this would break the reproducibility tests: this PR changes the behavior of the driver exclusively when there is a preexisting output folder where you are running a new simulations.

Ok, this makes sense, and I agree that this shouldn't break (most) reproducibility tests.

If you run the driver multiple times with the same configuration file, the same output folder is used. Data from different simulations is mixed, and NetCDF files from different simulations get appended to each others. For interactive work, this is really annoying.

Why is data from different simulations mixed? And why are NC files from different simulations appended? Can't we instead change that logic? This workflow is equally annoying to me: running a simulation 10x interactively will generate 10x the amount of data compared to just overwriting old files. Isn't that what the restart flag is for?

New data is appended because that's how the NetCDF writer works: if there's already a file (e.g., after the first time you output), you new data there. So, overwriting would mean cleaning the output folder before the new run. However, I strongly believe we should never erase existing information.

We can compromise and add an option to either overwrite or move existing folders, but the default should be to move.

[szy21]

Maybe I can talk about my other idea on this: I think we can add simulation information to the output file name, especially after we update the time type. So the file name would be something like ts_1d_average_19790101-19791231.nc instead of ts_1d_average.nc. If restarting, the new file would be something like ts_1d_average_19800101-19801231.nc, so no overwriting. If the output file name is the same, I think it's ok to overwrite, as that's usually what the users want when rerunning a simulation. I was thinking we can do this later though.

[Sbozzolo]

Maybe I can talk about my other idea on this: I think we can add simulation information to the output file name, especially after we update the time type. So the file name would be something like ts_1d_average_19790101-19791231.nc instead of ts_1d_average.nc. If restarting, the new file would be something like ts_1d_average_19800101-19801231.nc, so no overwriting. If the output file name is the same, I think it's ok to overwrite, as that's usually what the users want when rerunning a simulation. I was thinking we can do this later though.

Adding the dates is a little tricky. should the filename always match the content (I think it should). So, should we rename the file every time we add new data? What happens if the run ends before the end (for a crash, or because it was gracefully exited)?

I think we can add start dates because it is always well defined and not problematic. We could also create maybe subfolders (e.g., one per start-date) for ease of navigation.

In any case, I stand by the good user-experience principle that a program should never remove users' files without explicit consent (but we can add a --overwrite flag).

[charleskawczynski]

We discussed this today: @Sbozzolo is going to update the logic to move the folder to one with an appended counter, so that the output folder (after restarting) is easier to predict. This will allow us to test restarted simulations and know where the data will end up much more easily.

[Sbozzolo]

I implemented the logic to handle this in a module CliMA/ClimaUtilities.jl#28 (so that it can be used by Land and Coupler too).

The OutputPathGenerator module offers different styles for preparing the directory structure for the simulation output. The one I am using in ClimaAtmos is called ActiveLinkStyle and is approximately what is discussed in the previous message.

With the ActiveLinkStyle, files are saved in a subfolder of output_path, with the latest output always available in output_path/output-active.

Example:

Suppose we run a simulation with output_path abc. The first time the simulation is run, a subfolder abc/output-0000 is created and files are saved there. A link abc/output-active -> abc/output-0000 is created too so that the output path is predicatable (always abc/output-active). The second time the simulation is run, files are saved to abc/output-0001, and the link is moved to abc/output-active -> abc/output-0001. Once again, the output is predicatbly found in abc/output-active.

[Sbozzolo]

If there are no more comments, tomorrow I will merge the PR in ClimaUtilities and merge this too.

[szy21]

Thanks!

[charleskawczynski]

Looks fine to me