OpticsDescription table is inconsistent (and not readable)

[kosack]

Writing an optics table with subarray.to_table(kind="optics").write(output) writes a table in a format that is not readable by OpticsDescription.from_name().

It seems the format was updated in one place, but not the other. This leads to the files written by ctapipe-stage1-process to be not usable to construct a SubarrayDescription.

With other tables, like CameraGeometry we were careful to set a TAB_VER (table version) header keywords, so we could support data model changes, but it seems that was not done here.

Suggestion: we will have to add TAB_VER to this, but to support older version of the data format (which unfortunately are in ctapipe-extra and all output from ctapipe-stage1-process v0.8), we will have to make some code that reads both type (actually it seems 3 separate data models are assumed).

[kosack]

see #1358

[LukasNickel]

Are you going to add a second (newer) table to ctapipe-extra? Or what would be the next steps on this?

[maxnoe]

This should be fixed by #1405, right?

[kosack]

No I think it's a fundamental problem in the OpticsDescription to_table and from_table methods. They just need to be made consistent. There is a bit of a hack in #1405 to make it work, but we are definitely missing a unit test and having these two methods work without any hacks. The problem is that the file format changed in #982, but only the write method was updated, I think.

[maxnoe]

Ok, so we also need to update a file in ctapipe-extra, right?

[kosack]

Yes, or also support the old format (by looking at the TAB_VER header in the file). Might be a good example for what happens when we update the data model or file format in the future...

[maxnoe]

Fixed by merging #1417