Require a package's own tests to pass before allowing a version bump

[quinnj]

As discussed at JuliaCon 2015 Thursday evening:

• To bump a version of a package, a PR would be submitted, a bot/machine would download the proposed version of the package along with the proposed compatible version of julia and run the tests of the package
• On passing tests, the PR would be automatically merged and version bumped
• On failing tests, PR is not merged (locked?) and PR submitter is notified of failing tests or version mismatch

On Julia version bumps (minor or major), all packages would be re-run if the package has no upper limit on julia version compatibility to ensure that its tests still pass:

• If tests still pass, a new version is automatically bumped pointing to the same previous version
• On failing, the previous version is amended with an upper limit on the previous working version of julia; trying to Pkg.add() that package on the newer julia version would result in a no version compatible with your julia version exists or something like that

Obviously this is a lot of buildbot/testing infrastructure, but it seems like this would allow the state of METADATA packages to be 100% passing their own tests, with full transparency if changes to julia occur that break the package. The ideal goal here is that a user would never be able to Pkg.add() and get a package that "doesn't work" (at least according ot their own tests).

A related issue to discuss would be:

• minimum package testing standards that would at least be "eye-balled"

[timholy]

@jiahao pointed me to this. There's a lot to like about this proposal. The only problem I see with it is that I occasionally find myself reluctant to comment out tests that are failing for reasons "beyond my control," by which I mean highly-nontrivial upstream bugs that do not affect any of the platforms I depend on (like this one). I nevertheless sometimes would prefer to leave the test there to deliberately annoy myself. But maybe I just need to dig into more flexible testing frameworks?

[milktrader]

If you use the FactCheck package, you can use @pending instead of commenting out tests. This doesn't fail the tests but it does remind you that there is a test that needs to be dealt with.

[StefanKarpinski]

We should add that feature to the base testing support. It should not error but print a loud warning.

[tkelman]

This issue could use more automation. Right now we look over new packages in a pretty arbitrary way, depending on how deeply whoever looks at a proposed package feels like checking. Expanding either the checks on Travis or using an external service to do some quick checks, does your package have a REQUIRE file, LICENSE file, can it be added successfully, does it have unregistered dependencies, does it pass tests, does the case of the top filename match the package and module name, etc. We could start making Travis fail for these, but I think it would almost be better to have a bot leave comments with a link or summary of a package-lint report. Fedora and Debian have lots of little warnings when you make a package, and they're usually right pointing out things you should be fixing.

[sbromberger]

👍 for this idea.

[DrKrar]

I think the idea of just leaving a comment to the owner is a good practice.

[Nosferican]

This is moot and can be closed.

[DilumAluthge]

METADATA.jl is no longer maintained, so I am closing this issue.

If this issue is still relevant, please re-open a new issue on the Julia General registry: https://github.com/JuliaRegistries/General