Badging for project lifecycle

[cboettig]

I greatly enjoy using the use_* commands to get the markdown for my README badges. Curious what your team thinks of the http://www.repostatus.org/ badges, and if you would consider a l'lil function to generate the appropriate badge. I know there's some risk of these becoming stale since they are just static badges, but personally I find the categories quite helpful, particularly on the sunset era of a package.

[hadley]

I like the idea, but I don't love repostatus - it just feels a bit too fine grained to me.

Another option is https://github.com/badges/stability-badges but they don't feel quite right for the R community.

[cderv]

Just for pointing other interesting links. Here is an initiative for stability badges for R : https://github.com/joethorley/stability-badges based on shield.io services.

[cboettig]

The stability-badges makes sense to me once I realized I should think of them as package lifecycle terms, but I think stability is the wrong term. I see the motivation: e.g. plyr would be Locked, in the sense that commits would only patch bugs; perhaps closest to what repostatus calls Inactive, and unstable is approximately WIP, e.g. pre-CRAN release.

Yet when I first looked at stability-badges, I was greatly confused because I was not thinking of the in lifecycle terms but just as a generic indicator of the API. I think the unstable & stable are the most confusing: on the face of it, it sounds like stable is a pledge not never to break the API and never to bump the major version. Closer inspection shows that it does not make that claim, but then it becomes a bit unclear how this differs from an unstable package -- is an unstable package just a stable package that fails to bump the major version when it makes such a change? (This also gnaws at my dislike of semantic versioning -- sure there are big changes and small changes, changes that do or don't alter the API; but any change can break things, and I think the promise that "these changes won't hurt" is misleading or dangerous).

Focusing on the term stability, using unstable or stable, in my opinion, puts too much emphasis on classifying how the API changes and makes it easy to miss the lifecycle aspect (e.g. it is easy to miss the implication that stable is a more mature version of unstable, and instead think this is just a message of how much the maintainer gives a crap about the idea of backwards compatibility.).

If the those badges were called lifecycle badges instead of stability badges, maybe replacing unstable with the more life-cycle focused term beta and locked with a term that better indicates "this is an old package that gets limited maintenance priority / only 'security' updates / bug patches" I'd feel much better about it...

[hadley]

Ooh yes, I love the idea of focussing on the lifecycle!

[hadley]

Thoughts on how I see the life cycle of an R package:

• Experimental - very early days, a lot of churn in search of a good API (e.g. monads, strict). Not on CRAN. Might not go anywhere.

• Maturing - will strive to maintain backward compatibility (e.g. rlang, pkgdown), but need wider usage in order to get more feedback.

• Dormant - not currently working on it, (e.g. ggvis) but plan to come back to in the future.

• Stable - we're happy with the API (e.g. ggplot2). There are unlikely to be major changes, and backward incompatible changes will only be made if absolutely critical.

• Questioning - no long convinced this is a good approach.

• Retired - better replacement elsewhere (e.g. reshape, reshape2, lazyeval, pryr). Still on CRAN, and will fix R CMD check failures to keep it there.

• Archived - development complete. Archived from CRAN; github issues turned off.

And in a figure with most common transitions marked:

This lifecycle applies not just to packages but to components within packages. For example, the big pieces of dplyr (mutate, select etc) are stable, but I'm questioning do() in favour of list columns, and we're retired the SE version of functions. So as well as package badges, we might want roxygen2 infrastructure so that you can badge individual functions (e.g. @lifecycle mature would copy badge image into man/figures then generate embed linked image).

Note that it's particularly hard to know when a package is "stable" because it's hard to know when you're "done" with a problem. Sometimes it's straightforward (e.g. with smaller packages like forcats/stringr), but for bigger packages like ggplot2/dplyr they might be maturing for 5+ years, and they only become stable once I move on to the next generation of the tool.

[joethorley]

Some random thoughts

This is a good summary of a developers perspective. As a user stumbling upon a package I want to be able to quickly assess how stable it is and how likely it is to be supported long-term. I understand that stability is a continuum but its still important to get an indication of the frequency of and severity of breaking changes. And I want to know if the developer is committed in the long-term. These two pieces of information allow me to decide whether I am going to use the package (which also depends on how important it is to my work)

The categories above can be classified in this way

• experimental: unstable and may be short-lived (use if inspired)
• maturing: more stable and looking like it may have longer shelf life (use if directly relevant to workflow)
• stable: stable and long-term support (use)
• questioning: stable but may not have long-term support (use if no better alternatives)
• retired: stable but no longer long-term support (don't start using and transition to alternatives)
• archive: so stable changes in other packages are going to break it with no long-term support (stop using)
• dormant: unstable and no long-term-support (use if willing to take on)

I think the schema at https://github.com/joethorley/stability-badges is reasonable but could be improved - its good to see the topic getting attention.

I also think dormant and questioning are unnecessary categories for the end user because a) both are potentially reversible; b) dormant can be inferred from the commit history and c) its not until a package is retired that a user needs to switch.

If these are dropped one is left with a linear sequence of

experimental, maturing, stable, retired and archive

these roughly correspond to the five categories in https://github.com/joethorley/stability-badges of

Experimental, Unstable, Stable, Locked, Deprecated

I struggled with the naming scheme and definitely prefer Maturing over Unstable. However I prefer Locked to Retired since its clearer what the change is. Archived seems superior to Deprecated.

I also don't like the references to CRAN since a whole package can live out its lifecycle on GitHub.

Hopefully these reflections have some value.

[mwaldstein]

I agree about the difference between developer view and user view and want to amplify that idea.

The model outlined here is partially compatible with semantic versioning (http://semver.org/). One challenge R faces that the js world has been rapidly iterating around is the idea that you don't always need to be on the latest version - once you have a dependency that is stable for your use case, don't upgrade unless there is a convincing reason. R tooling doesn't really support this approach - e.g. stay on 1.0 rather than upgrade to 2.0 because it still works. In the context of lifecycle, this means that "stable" just means "the current design is well tested and has some guarantee of working" not "the api won't change".

I feel w/in this discussion, more valuable is a communication around level of support and iteration as @joethorley suggests - e.g. the range between "Wild ideas welcome" to "If it works for you, great, if not fork it b.c. I'm no longer supporting"

[cboettig]

@joethorley Great feedback, thanks for sharing. I agree picking the right adjectives is definitely tricky. A few quick thoughts:

• re: dormant: I think this is here to distinguish from retired. Both would have a quiet commit history. What badge would you suggest using for, say, ggvis in Hadley's example if not dormant? (Obviously leaving it unbadged is also ambiguous).

Re developer vs user perspective: I think the concept of lifecycle is actually more about communicating with users than developers. I believe the concept of API stability is primarily a concern only of relatively sophisticated users (aka other developers who use or depend on the package. Not saying users shouldn't care about API stability, but that is not a significant factor in how I see most R users selecting packages). For instance, many R users will simply try to interpret these badges in a hierarchy, where it is easy to think "Locked > Stable > Experimental", and I don't think most R users would be better served choosing plyr because it is "locked" stability! yay! over dplyr which is merely stable, say. It takes a developer's mindset to realize that the rock-solid stability of "locked" really means "old package replaced by something better".

Re semver, I agree with @joethorley 's perspective that semver is really focused on packages in the stable bucket as way to signal to developers the nature of changes being made within versions of an individual package. It doesn't speak to the issue of whether I should use plyr or dplyr, or let me know that pkgdown API is likely to change.

Re references to CRAN: I agree the syntax should be CRAN agnostic and R agnostic, but "not yet on CRAN", "on CRAN", and "archived from CRAN" are currently the closest thing we have to functional lifecycle signposts in the R community, so making the analogy explicit is helpful.

[joethorley]

@cboettig I would suggest using Experimental for ggvis because they only way it differs from an experimental package is that it is not currently being worked on (as indicated by a quiet commit history). If it was retired it could be badged Retired.

You are right Locked might be misleading for less advanced users and therefore Retired might be better. The question of how to interpret the terms is partly dealt with by clicking on the badges which can lead to a definition of what the term means. I also think that a short page could explain to users what the five or so badges mean and how to relate to them.

[hadley]

Re: questioning - this is very important for me (albeit more at the function level than the package level). I need some way to communicate to people that I think there's likely to be a better approach in the future and not to invest heavily in the current way of doing things.

Re: dormant - I don't think it's reasonable to expect to carefully examine the commit history to determine the repository status. By chance, there may have been a brief flurry of activity on a retired package because a patch was needed. Dormant explicitly says that we're not working on this right now, but we expect to return to it in the future. I don't believe that the life cycle of a package is a linear sequence.

Re: CRAN - I think it's important to start out concrete and specific before generalising. Specifically to R, I would never depend on any package that doesn't have a long term plan to get on CRAN. Not aiming for CRAN suggests that the developer doesn't particularly care about the package (in the same way not using unit tests suggests that they're not very committed)

Re: semver - I think it's naive because there's no way to tell what is "backwards-compatible bug fix" without knowing all user code. I've fixed plenty of bugs that have broken analysis code because people were dependent on the broken behaviour. To me, this seems like a large unacknowledged flaw and makes it very difficult for me to consider semver as a realistic proposal.

[hadley]

And here's what the badges might look like:

[joethorley]

I like the colouring scheme but would suggest dormant should be orange because it exists between experimental and retired which are both orange and because it should not be rated as highly as maturing or questioning.

[hadley]

Maybe that's because the arrows are in the wrong place - might be better for them to go to/from maturing.

[gmbecker]

Interesting discussion. It seems like a couple different things may be being conflated here where I'm not sure we want them to be. One is the concept of "published" vs "development" versions of packages.

If a package is being rapidly developed on github, but there is a stable(ish) version on CRAN whose changes are slower, I would call that package (the CRAN version) stable. The fact that the unpublished development version of dplyr, for example, might be rapidly (and disruptively) changing in the preparation for a new major version in the future doesn't change the stability of the published version as it is now.

I guess what I'm saying is that I'm having trouble imagine these badges applying to a package as a singular entity, without taking into account which "flavor" of the package you're talking about, at least in many cases.

I also don't agree fully with what @hadley and @cboettig seem to be saying about semver, because in some contexts I'm not convinced behavior changes are synonymous with breakage in a couple of different cases. Maybe that's a topic for a different forum, but just briefly, and IMHO, API-breaking changes are fundamentally a different class of thing than narrow fixes for (specific) bug. Obviously bug fixes must change behavior in certain cases, because if they don't there either wasn't a bug or it didn't get fixed, but that doesn't make them API breakages. API changes mean the contract with users has been changed, bug fixes mean that the contract wasn't being met before, and now it is.

[hadley]

I don't see the conflation of CRAN vs dev versions, except that some states (experimental & archived) don't appear on CRAN. I'm thinking that these states generally apply to the released version of the package (because that's what people actually use).

I'd prefer not to sidetrack the discussion here with any more thoughts on semver.

[hadley]

Note to self: this might also be useful for labelling book chapters

[gmbecker]

@hadley In my mind a development version of a package is ALWAYS experimental, even if there is a stable version on CRAN, thus the potential conflation. Particularly if the badges are showing up on github repos where the devel versions typically live.

As for the semver thing I really don't care if people follow the actual semver specification or not (I think I probably don't but I'd honestly have to go back and look at exactly what it calls for to be sure). My point was that not all changes are created equally, and that stability as its typically used in software AFAIK is about lack of disruptive API breaking changes, and says very little about the frequency of narrow bug fix style changes. I think that is what stable should mean here too. If something is actively maintained with a lot of users it is going to have bug reports and bug fixes, but I don't think that should preclude it from having the "stable" badge. @cboettig seemed to be disagreeing with this in a previous comment, and I was looking to get some conversation started about that to see if we could land on what people generally think stable means before we start badging things as stable.

[hadley]

I broadly agree your definition of stable - in a stable package you will strive not to make breaking changes to the API, but you'll continue to fix bugs (even though a user might not be able to tell the difference). although sometimes a package can be largely stable but still have a few parts of the API that are evolving.

[joethorley]

This is what I consider to be stable: Backward incompatible changes will be restricted to major releases (if possible!) and functions and arguments will be gradually deprecated with informative warning messages.

[gmbecker]

@joethorley I agree as well. I think what @hadley is getting at is actually something else, which is the lack of a formal, broadly used mirror concept for deprecated for experimental features that should not be considered part of the API when assessing stability of the project as a whole.

One idea in the R space is just to have a .Experimental() function analogous to the existing .Deprecated() function. i.e. you call it in the beginning of your function and it informs the user that they are using an experimental feature within your package (the implication being that they cannot rely on that particular part of the API to be stable). I think I may be getting a bit far afield of the discussion at hand though, so I'll just leave it at that for now.

[hadley]

I also don't think limiting backward incompatible changes to major versions actually has much impact on R users. It's good practice, but most R users just type install.packages("blah") or update.packages(); there's no easy way to deliberately choose minor updates.

[hadley]

@gmbecker btw I've read more around semver, and now understand the situation better. The problem of bug fixes sometimes being API changes for some people is considered in the broader community discussion, it's just not part of http://semver.org.

Generally, I think we should strive for limiting API changes to major releases, while accepting that non-API changes can break code, and that R currently lacks good tools for pinning to major versions.

[hadley]

Plan is to build tooling (like .Experimental) in what's currently called oldie but will eventually be called lifecycle.

[jsta]

These badges are great! Could the badges be made to link to the definition of each lifecycle stage rather than the standalone svg?

[jennybc]

@jsta Will you open a new issue for this? I see your point. We need to figure out where to park said information, so we can then link to it.

[cboettig]

Just for future reference, looks like badges now link to https://www.tidyverse.org/lifecycle/ .

Very slick and thanks @jennybc & @hadley ! Looking forward to seeing these in the wild (cough, on tidyverse & r-lib packages!)

[batpigandme]

@cboettig in the works…