Refactor the "exclude" feature

[danielhuppmann]

All validation methods have a flag "exclude_on_fail" feature to easily track which scenarios fail one or several validation steps. This is implemented via an "exclude" column in the meta indicators.

However, this approach causes some hiccups in the integration with the ixmp database infrastructure - the "exclude" meta indicator is always added by pyam and then has to be removed before importing to the database (or can cause confusion by users).

My proposal: drop the "exclude" column from the "meta" dataframe and move it to an own attribute. in addition, extend the filter() method such that df.filter(exclude=True) continues to work as is.

Any thoughts @phackstock @gidden @byersiiasa?

[gidden]

I always end up making my own meta indicator anyway - so I agree with your proposal to remove

…

On Wed, Jun 28, 2023 at 9:59 AM Daniel Huppmann ***@***.***> wrote: All validation methods have a flag "exclude_on_fail" feature to easily track which scenarios fail one or several validation steps. This is implemented via an "exclude" column in the meta indicators. However, this approach causes some hiccups in the integration with the ixmp database infrastructure - the "exclude" meta indicator is always added by pyam and then has to be removed before importing to the database (or can cause confusion by users). My proposal: drop the "exclude" column from the "meta" dataframe and move it to an own attribute. in addition, extend the filter() method such that df.filter(exclude=True) continues to work as is. Any thoughts @phackstock <https://github.com/phackstock> @gidden <https://github.com/gidden> @byersiiasa <https://github.com/byersiiasa>? — Reply to this email directly, view it on GitHub <#755>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKUAEN5JNHUDMRZSWDW2I3XNPP7LANCNFSM6AAAAAAZWUVVRI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[byersiiasa]

Sure - sounds fine to me.
But would there be a method to edit the exclude attribute manually?
e.g., perhaps similar to how one might use set_meta('exclude'...)