Clarification/Documentation of policy for deprecating packages

[kmsquire]

There are two two issues I would like to see clarified:

1) sha1 bumping should (almost) always accompany a new version (with exception of version 0.0.0). (Does current policy allow bumping of any x.x.0 version? Should this only be in effect for released versions of Julia?)

Rationale: no matter how careful one is, some new version will break on someone's system, and it's important to allow the user to revert to a previous version. Without bumping the version, this is hard for the naive user.

2. Old versions of packages should (almost) never removed from METADATA.jl

Rationale: Removing a package version from METADATA.jl causes problems when 1) a package depends on a removed version of a package, or 2) a user wants to replicate some results produced with a particular version of a package, and that version is removed.

---

My impression is that these are policy, or at least should be, but I don't think they're documented, and both have been violated

Any suggestions/objections/refinements? Should I file an additional issue for Pkg2 to support updating packages?

[IainNZ]

The thing thats been confusing me about 2) is that some of those versions probably won't work with any point release version of Julia - they worked for HEAD julia at some point, but no longer. Surely its OK to remove those?

[johnmyleswhite]

I agree with Iain's concern: in order for us to really support any old packages, we would need to depend upon specific SHA1's of Julia. Almost all of my julia 0.2- codebases wouldn't work on very old versions of Julia after 0.1 was released.

Although it clearly sucks, it's not clear to me that we can viably start being fastidious with versions of Julia until Julia development slows down to the point at which it's reasonable to encourage somewhere to keep using Julia 0.2.2 for 12+ months.

[kmsquire]

Revisiting this issue... 1) is already handled. For 2):

Currently removing or changing the name of packages is not a good idea, currently, as we saw with Stats.jl(*).

A proposal moving forward:

1. All packages should specify a minimum version of Julia they work with
   • @IainNZ, I think you proposed this previously
2. To remove a package
   • Set the last Julia release it should be used with
   • Tag it
   • Transfer package ownership to the JuliaRetiredPackages group (to be created), and update the METADATA.jl url accordingly
3. To rename a package, follow the steps in Named: 0.1.0 #2, and then
   • Rename the package (after transferring ownership to JuliaRetiredPackages)
   • If desired, delete any git tags corresponding to version numbers
     • Alternatively, set the version number appropriately when adding to METADATA.jl
   • Add the newly renamed package to METADATA.jl
4. Periodically create a new METADATA.jl branch, removing packages which do not work with recent versions of Julia
   • Older Julia versions would point to other branches (which would not get package updates)
   • The default branch to use for each Julia version could be stored in METADATA.jl itself (in all branches)

I think these steps would alleviate future problems in renaming/deleting packages. It's likely that most of this functionality could be added to Pkg (e.g., with Pkg.deprecate() and Pkg.rename_package() functions).

Looking for thoughts and feedback.

Cc: @StefanKarpinski

(*) In particular, one of the problems with Stats.jl was that .julia/.cache (or .julia/v0.x/.cache) still contained a copy of the unrenamed repository, which clashed with the new Stats.jl repository that was created, since they did not have the same git history.

[IainNZ]

I think (1) is so easy to do, and no downside - its a minimum version, not a maximum.

[IainNZ]

This is a real issue, see ulysses4ever/Monads.jl#4
Would be good to have an official policy blessed by a higher power

[jiahao]

I think what happened in #2346 is an extreme example where the right thing to do really is to scrub the entirre record from METADATA.

[hayd]

Is METADATA supporting 0.2 (or -) users indefinitely??

Perhaps similar to @IainNZ's great work on the pkg tests, there could be a package health heuristic for automatically putting "unhealthy" packages up for deprecation consideration. E.g.

• recent release
• tests pass (supports most recent julia release)
• has documentation
• recent git activity
• latest git issue(s) responded to

If these were all/mostly bad then first ping the maintainers to fix things, and if the health does not improve then flag for the community to consider removal.

Some packages may fail some of the above but needn't be removed, e.g. they are important or failing X is no big deal, or it may be they can be moved (e.g. to a julia org).

[wildart]

As issue PkgDev/#21 suggested, there should be some functionality to deregister/unmaintain/unlist registered packages.

I propose to add unlist empty file to the root of a package folder which would indicate that the particular package should not be installed/used. If developer will decide to resume development this file will be deleted.

All necessary functionality will be provided by PkgDev.

[IainNZ]

Whats wrong with the current approach?

[wildart]

Zeroing version disables only a particular version of package on which this operation was performed. If package has multiple versions, then other versions can be accessed, unless you set requirements to zero in every version.

[IainNZ]

The current approach is to set a Julia version cap for every version

[wildart]

Ok, I though only last version get updated. Anyway, what if package is needed to be restored, then the version requirement information is lost. Well, not completely, it's in METADATA commits, and a new version of the package must be issued anyway. I am just weighing arguments. Current, solution was developed because, we needed to "deregister" packages without changes in work of the package manager. By adding additional information to the package registration we could warn user about installing an old & unmaintained package (e.g. add unmaintan file) or completely deregister (e.g. add unlist file) a package.

[tkelman]

Deregistering is potentially dangerous if anything depends on the package in question yet. You're aware of last week's npm left-pad fiasco, right?

The only thing needed to restore a deprecated package is tag a new version that supports up-to-date julia versions. PkgDev.deprecate would make sense to formalize and automate the existing pattern of adding an upper bound to all past tags.

[wildart]

I would not worry about deprecation so match, cause we have screening process when merge any PR.

As for zeroing all versions in package, it looks as a drastic measure. Adding additional info to package description fills less invasive then changing content of multiple files. Plus, we can get different and more verbose messages on various pieces of information, rather then saying "not available".

[tkelman]

Except that "package description" does not formally exist in current package manager functionality. It could if JuliaLang/julia#11955 gets implemented (which would also allow more specific error messages as you suggest), but I'd call that more invasive than bounding multiple tags in metadata. Nothing is zeroed, just add an upper julia version bound.

[wildart]

Adding upper bound does not guaranty that package would not be used, or would not inform user that the package is unmaintained.

[tkelman]

It will no longer be possible to install with Pkg.add. That's good enough for me. Of course a note in the readme would be ideal, but when a package is abandoned there's often not much we can do about it.

[wildart]

I see it as a problem, what if a package is functional, but a developer decides that he will not going to maintain it. Why would this package be prohibited from installation? Wouldn't it be better to mark a package as unmaintained and to inform user, which installs it, about that?

[tkelman]

That could be accomplished with a warn on using, couldn't it? There's no mechanism in Pkg for that kind of thing right now, and as far as things we could use 11955 to accomplish this seems like it would be low on the list.

[Nosferican]

There are more packages that started including parsable metadata such as repo status. Could part of that information be included in the metadata and displayed during installation or changes during updating?

For obsolete packages, a sensible option would be to allow it as long as it passes its test for the Julia version the user is using when installing it (inferred upper limit). The other case would be a depreciated package which should be replaced. Maybe releasing a version for updating the metadata with a depreciated use X package instead tag.

[DilumAluthge]

METADATA.jl is no longer maintained, so I am closing this issue.

If this issue is still relevant, please re-open a new issue on the Julia General registry: https://github.com/JuliaRegistries/General