You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Saving the ensemble to parquet would be useful for saving state. The object and source tables should probably be saved as two separate parquet files.
Neccessary Components:
ensemble.save_ensemble() function; Saves an Ensemble to disk, with subdirectories for object and source partitions. Option (perhaps on by default) to include all result frames, or just a subset of result frames
ensemble.from_ensemble() function; Reads the directory structure established by save (parquet-like) and loads into a new ensemble, I think this will need to be a separate function from read_parquet to handle the addition of many result tables implicitly
from_ensemble(); An Ensemble constructor function that bypasses ensemble initialization
unit tests; with some consideration to how we test the save function within the actions framework. Does it just purge any created files automatically? Edit: No it doesn't. Resolved this by using pytest with pathlib.Path temporary directories
documentation; this feature will be important enough to need some documentation beyond just autoapi.
Non-critical Components:
Potentially save the column mapper and load it within the save/load internals. Avoid needing to specify the column mapper in the load.
Open Questions/Issues:
Schema error with saving Stetson_J result frames to parquet. These frames have a dictionary result column, it may be that there's a simple schema argument to pass, but this would also be resolved by just making stetson_J return a column per band (as has been talked about for a while). Given that stetson_J is really just a tester function for us, I'm opting to not spend time on getting it working.
The text was updated successfully, but these errors were encountered:
Saving the ensemble to parquet would be useful for saving state. The object and source tables should probably be saved as two separate parquet files.
Neccessary Components:
Non-critical Components:
Open Questions/Issues:
The text was updated successfully, but these errors were encountered: