Robustness evaluation [API]

[LukeWood]

No description provided.

[bhack]

What is this?

[LukeWood]

Sorry this should not have had the contribution welcome label. I will post a more thorough explanation when it is ready. I try to use GitHub for all things KerasCV - this includes notekeeping.

[LukeWood]

Basically @bhack the goal of this is to go from model and dataset -> a table telling you how robust your model is to various perturbations including blurs/gradient attacks/etc

[bhack]

Are these a sort of ablation studies?
Do you want to use KLPs on the testset as a sort of adversarial?

[bhack]

Or something more complete like:

https://github.com/Trusted-AI/adversarial-robustness-toolbox

[bhack]

A fresh Deepmind paper:
A Fine-Grained Analysis on Distribution Shift

[bhack]

https://shift-happens-benchmark.github.io/

https://github.com/shift-happens-benchmark/icml-2022

[LukeWood]

The KerasCV team is attempting to narrow the package scope in order to more efficiently use our limited resources. As such, we are only accepting contributions directly related to items on the 6 month roadmap for the package.

If you feel this feature could be particularly useful, or would like to contribute it - please feel free to re-open this issue and ping LukeWood, ianstenbit or tanzhenyu

Feel free to read more on the roadmap:
https://github.com/keras-team/keras-cv/blob/master/.github/ROADMAP.md