Dependabot's compatiblity score needs to be improved

[ap-wtioit]

With ypasmk/robot-framework-docker#13 dependabot gave a 100% compatiblity score for upgrading pillow 6.2.0 to 7.1.0 for a python2 project.

However pillow 7.0.0 states:

Python 2.7

Pillow has dropped support for Python 2.7, which reached end-of-life on 2020-01-01.

https://pillow.readthedocs.io/en/stable/releasenotes/7.0.0.html#backwards-incompatible-changes

So dependabot should have choosen 6.2.2 instead of 7.1.0 for python2

Fixed broken build with ypasmk/robot-framework-docker#15

[jeffwidman]

Oof, that's clearly not compatible!

Looking at this deeper, a few things happened:

1. When I look at https://github.com/ypasmk/robot-framework-docker, I don't see a way that Dependabot can ascertain whether the project is a python 2 or python 3 project... the only file it has from a python perspective is requirements.txt. So we can easily parse the Trove classifiers for pillow, but we don't have a straightforward way to know what your project supports. The only way would be is if you treated it as a library and included a pyproject.toml that stated the supported python versions.
2. Compatibility scores are based on CI runs across all repos that are doing this upgrade. So in this case, probably all the other repos that were doing this upgrade were already python 3 compatible, so they returned 100%.

I totally get that this was a frustrating user experience, but I just don't know what we could have done differently here to prevent this in a programmatic way.

[ap-wtioit]

Things i can think of how you could improve for the compatibility score:

• Be open about which CI runs contributing to the compatiblity score (this way one could compare if those projects are similliar) instead of linking to the not very helpful help page
• Add the other dependencies from requirements.txt into the compatiblity score calculation. There is lots of python packages that have dependencies with others. Updating just one often doesn't work as intended.

[jeffwidman]

Definitely agree that there's a lot of room for product improvements with the compatibility score. For example #2250.

Unfortunately, right now there's a bunch of other things that are higher priority--both here in this repo and also behind-the-scenes work we're doing on the infra.

Once those are handled, then I suspect we'll start to look at how we can leverage the data of running millions of updates across all these repos to improve the lives of our users and also library maintainers.