Add multilayer preproc

[msilvafe]

Addresses issue #1003.

[Wuhyun]

I ran it successfully on two-step preprocessing, split at demodulation. Looks functional to me - thank you for implementing this!

Test performed (found under NERSC /global/cfs/cdirs/sobs/users/wuhyun/multilayer_preproc_tests for anyone interested):

# 1) Initial preproc
preprocess_tod(obs_id, config_1, group_list=[["ws0", "f150"]],...)
# 2) Dependent preproc
multilayer_preprocess_tod(obs_id, config_1, config_2, group_list=[["ws1", "f150"]],...)
# 3) Load from saved results
aman = multilayer_load_and_preprocess(obs_id, config_1, config_2, dets={"wafer_slot": "ws0", "wafer.bandpass": "f150"})

I have one general question. Currently, the initial preprocessing (preprocess_tod) needs to be run before the second (multilayer_preprocess_tod). Would it be worth combining the two processes (e.g. have multilayer run preprocess_tod inside it), so that we can save up some time reloading the first preprocess?

For context, I found that on NERSC login node it takes ~120s to run and ~60s to load a preprocess config up to demodulation (single wafer & bandpass). Maybe these run fast enough that we don't worry about it much?

[mmccrackan]

I have one general question. Currently, the initial preprocessing (preprocess_tod) needs to be run before the second (multilayer_preprocess_tod). Would it be worth combining the two processes (e.g. have multilayer run preprocess_tod inside it), so that we can save up some time reloading the first preprocess?

Thanks for testing!

I had considered incorporating preprocess_tod into multilayer_preprocess and it certainly is possible to do this, though there are a few considerations. There are some complexities due to the desire to run many obs in parallel and the need to do the outputs at the end. Since multilayer_preprocess_tod needs to check the pre-existing database from preprocess_tod, it would probably need to be accessed in parallel and populated for all entries that multilayer_preprocess needs.

We have plans to integrate a single main function interface for preprocess_tod, multilayer_preprocess_tod, and preprocess_obs in order to make using these functions simpler, so maybe we can revisit this after we have done that.

[Wuhyun]

Thank you for your answer, that sounds good to me.
Please add that one import statement and I'll be happy to approve this if others are.

[msilvafe]

Thank you for your answer, that sounds good to me. Please add that one import statement and I'll be happy to approve this if others are.

Hey @Wuhyun thanks for reviewing and testing! What is the import statement that you're referring to?

Also after chatting with @mmccrackan we're going to go ahead and implement your suggestion to allow the multilayer function to run through both database builds and not require that the first one is prebuilt via preprocess_tod.

[Wuhyun]

Sorry, forgot to submit this eariler.

[mmccrackan]

Okay, I have made a bunch of changes to the functions here. I've tested it a fair bit, but there are a lot of logic branches here, so I probably missed a few cases. The main differences are:

1. multilayer_preprocess_tod.py will now call preproc_or_load_group.py, meaning that it can run the pipeline on the first config if the db is not found or overwrite is true.
2. preproc_or_load_group.py has been refactored to allow for one or two config files. If running one, it will operate like it did before. If two, it will either load both or it will load or run the first config and then run the second config.
3. both preprocess_tod.py and multilayer_preprocess_tod.py now write out temp files for each group instead of just a single file for all groups. The former writes to "temp/" as before, whereas the latter writes to "temp_proc/".
4. cleanup_mandb is now used for merging all the temp files in both cases.
5. Some helper functions like find_db and save_groups have been added and are used throughout.
6. Since we're now matching the per-group files of preproc_or_load_group.py in preprocess_tod.py and multilayer_preprocess_tod.py, the outputs have changed. The destination files are no longer output separately, but are output inside of the output lists. So the outputs for preprocess_tod.py are error and outputs, whereas for multilayer_preprocess_tod.py they are error, outputs_init, outputs_proc. The outputs may be empty depending on the runtime configuration.

[msilvafe]

This logic should go before pipe.run if we want to catch the logical case where the code exited after writing some of the groups (but not all) to the temp directory.

[msilvafe]

In check_cfg_match if the dependency check fails it just raises a logger.warning() and returns False let's raise an exception or print something to stdout here in the else statement that states that the dependency check fails.

[msilvafe]

This should probably go to a logger instead of to stdout (if we actually need this, maybe only needed for debugging)

[msilvafe]

Not an argument.

[msilvafe]

context_<> arguments not used

[msilvafe]

wrong type of quotes should be `` at the beginning of multilayer_load_and_preprocess. If the db entry does not exists or not of

[msilvafe]

Same comment as line 484 here and at 583

[msilvafe]

write this (and line 618) into a try-except similar to lines 624/648 to prevent crashing @chervias's mapmaker script and writing a more useful error message to his error log.

[msilvafe]

These error messages should now include reference to whether this is error-ing out in the init or proc section to aid debugging where in this mess of nested if-else statements we're exiting.

[msilvafe]

This check should probably happen before getting to this step as referenced above in L642

[mmccrackan]

I've added some things to address the existing comments. There are likely still bugs or logic paths that might fail so I'll keep testing, but I wanted to get the updates out so everyone could take a look.

The biggest change is the addition of a new function save_group_and_cleanup which will look through all the input groups, find any for which a temporary file exists, call cleanup_mandb (creating a db if it doesn't exist), and delete the file afterward. If the file exists, but cannot be opened, it will also be deleted (in case of file corruption). This is intended to allow us to use the temp files from a run that failed before creating and populating the db and destination file for all groups. I pass through an overwrite option as remove (I changed the name to remove, since overwrite would imply the opposite behavior within save_group_and_cleanup) which will simply delete the files. This is added to allow for overwrite=True to work in preprocess_tod.py.

It is probably a good idea to do this before running preproc_or_load_group followed by cleanup_mandb. To do the same for one group, you can just do:

dets = {gb:gg for gb, gg in zip(group_by, group)}
outputs_grp = save_group(obs_id, configs, dets, context, subdir)

if os.path.exists(outputs_grp['temp_file']):
    try:
        if not remove: # overwrite
            cleanup_mandb(None, outputs_grp, configs, logger)
        else:
            # if we're overwriting, remove file so it will re-run
            os.remove(output_grp['temp_file'])
    except OSError as e:
        # remove if it can't be opened
        os.remove(output_grp['temp_file'])

I've also wrapped up most commands in preproc_or_load_group into try, except blocks, so hopefully nothing should result in an uncaught exception.