Add support for custom model card fields and schemas

[timlrx]

Hi @shuklak13, following up on our call last year :)

What would be the best way for us to manage our own schema that is built on Google's model card toolkit? As you know, we currently manage our own fork of the model card and are creating other tools and code to improve the editing process.

While we could just run with our own version of the model card and break the compatibility, I think it will be nice to maintain the backward compatibility so that our users can also use the base model card libraries e.g. for the tensorflow integration and existing Google model card users can make use of our web viewer.

Let me know if you have thought of a good way of managing it. Otherwise, currently for every change to the proto file that you make, we might have to make breaking changes to our code. Thanks.

[shuklak13]

Hi Tim!

Google's model card toolkit today doesn't extend to custom schemas very well. We are currently planning a 2.0 release of model card toolkit which will move away from enforcing the existing proto schema, and will use a LitElement-based frontend instead of Jinja. The goal is to have greater extensibility and flexibility to support a wider range of use cases like yours.

This is still in the planning phase so I can't promise a timeline, but this issue is on our roadmap. For now, I think maintaining a separate fork for VerifyML makes sense. Once I have a clearer idea of the Google Model Card Toolkit 2.0 roadmap/timeline, I'd love to reconnect and discuss how Google model cards can support Cylynx VerifyML.

[timlrx]

Sure, happy to reconnect again once Model Card Toolkit 2.0 is out. Thanks!

[stevespringett]

@shuklak13 Has any progress been made on v2.0? Other than extensibility that you mentioned earlier, I'm especially interested if the JSON schema in v2.0 will vary from the existing v0.0.2 schema in terms of the types of data represented in the model or how it's represented. For reference, the OWASP CycloneDX bill of materials standard will be supporting ML components in the next release of the spec. We'd like to align to the great work that's been going on here and make it as compatible as possible.

[codesue]

Hi all, Model Card Toolkit is now a community-led open source project under the TFX Addons special interest group. (Learn more in this announcement.) The project now depends on community contributions, bug fixes, and documentation. This means, for example, whether we can move from Jinja templates to a LitElement-based frontend depends on whether contributors from the community volunteer to implement, review, and maintain this code.

One idea for schema flexibility and extensibility is to use pydantic to dynamically generate schemas, add an option to set a custom schema version for serializing and deserializing model cards from JSON, and make it optional to output model cards as protos.

@stevespringett, #248 introduced a breaking change, so the next release will be to the next major version (2.0.0) to comply with SemVer. Currently, there's no work planned to change the existing v0.0.2 JSON schema for v2.x.

@timlrx, for maintaining a fork, in general, I'd encourage trying to upstream any changes you think might be useful for the broader community.

[skylarbpayne]

Thanks for the update @codesue . My team is currently evaluating using model-card-toolkit and may possibly want this functionality (and would be happy to contribute). I may be missing something simple, but is there a reason we couldn't just add an extension field to the ModelCard proto to let users define one more top level field?

Given the flexibility of layout w/ export (due to Jinja) this seems: (1) light weight and (2) backwards compatible and (3) covers probably 80%+ of use cases. LMK if I've missed something :)

[codesue]

Hi @skylarbpayne, I don't define protos regularly, so I'm not familiar with all the gotchas when working with them. That said, adding an extension field sounds reasonable to me, and I don't know of a good reason not to do it. And yay for a lightweight and backwards compatible solution! We'd be happy to receive your PR. 😄

[skylarbpayne]

I realize extension fields aren't supported in protobuf 3, only in 2. But looks like you may be able to get a similar implementation with a single top level field with type Any. Unclear about the level of support of Any in python bindings though.

We're still evaluating, and given the pain points with dependencies in this repo, it looks unlikely we'll adopt. But hopefully this little idea helps someone else!

Thanks!