Provide a clear definition of "Backlog" with guidelines for using it

[flyingzumwalt]

There was a fair about of discussion about the meaning of a "Backlog" in the PM pipeline and even more discussion about how to name sections within the backlog -- future work, icebox, ready, etc. I've gathered all of that info here.

Work to be Done

Update the PM document to provide a clear definition of "backlog" with

• a clear definition of the term backlog
• something that explains why this concept is important and declares how to be "consistent" with this model's usage of that term
• an example accompanied with a diagram
• guidelines for using the backlog
• suggestions for sections within the backlog and what to name them

Confused by the Definition

I'm confused by the definition currently in the PM Doc:

Backlog is the collection of all goals in a project. [...]

By that definition, the Backlog includes Goals that are In Progress and Done as well as the Future Work, Ready, etc. This doesn't fit with the way I see the term used in the rest of the document.

The document later uses the term "Backlog" to label both a phase and a stage in the same diagram, but the document doesn't define the term phase or explain how the notion of phases should be used to arrange Goals.

Example from go-ipfs

In applying this model to go-ipfs (see notes) @whyrusleeping pointed out that most projects will need to communicate where each Goal lies on the spectrum shown in the table below. Based on our discussion, we came up with some ideas about how we can use pipeline statuses and Milestones to express this info. I included that info in the table. The specific choice of names for these statuses is changeable, but the key ideas are important.

What you want to communicate	Status	Milestones
"This Goal was reported but maintainers haven't looked at it yet"	New/Inbox	No milestone or milestone without a date
"This Goal has been reviewed but needs more attention before it will be ready to work on"	Refining	Might have a milestone
"This Goal is well defined but not planned to be carried out at this moment. (ie. it doesn't have a priority or a deadline)"	Backlog	No milestone or milestone without a date
"This Goal has been reviewed and is ready to work on"	Ready	Might have a milestone
"This Goal is ready to work on and we will be working on it soon"	Ready	Has a milestone with a date

This has a lot of overlap with these definitions in the draft doc:

Future Work - the goal is identified and being fleshed out. It is not ready to be carried out at this moment.
Backlog - the goal is well defined but not planned to be carried out at this moment.
Ready - the goal is planned to be carried out, waiting to be worked on.

... but note that we reconsidered the terms "Future Work" and "Backlog" in these comments

What is the goal here?

What are the consistencies we are trying to achieve with the term "Backlog"? If we're establishing this model in order to have consistency of language and core structures across our projects, what are we asking people to be consistent about?

This diagram from the draft PM document presents "Backlog" as both a stage and a phase, where the Backlog phase includes the Backlog stage and some other stages. Is that intentional? Is that going to confuse people? If we just need a different name for the phase, someone taking a scrum approach might label the "Backlog" phase "Refining" or "Ideation".

  ╭─────────────────────────────────── Pipeline ──────────────────────────────────────╮
  │────── 1. ────│──── 2. ───│─── 3. ──│────── 4. ─────│─── 5. ───│──── 6. ───│── 7. ─│
  │ Stages                                                                            │
                                                ╭────────────╮
──> Future Work ──> Backlog ──> Ready ──> In Progress ──> Review ──> Release ──> Done ──>
         │             │          │             │            │ 
         ╰─────────────╯          │             │            │
                                  ╰───────── Blocked ────────╯
    Phases
  │───────── Backlog ────────│──────────── In Progress ───────────│──── Complete ─────│
  ╰───────────────────────────────────────────────────────────────────────────────────╯

The document defines the stages with this key:

Future Work - the goal is identified and being fleshed out. It is not ready to be carried out at this moment.
Backlog - the goal is well defined but not planned to be carried out at this moment.
Ready - the goal is planned to be carried out, waiting to be worked on.
In Progress - the goal is actively being worked on, this moment.
Review - the development of the goal is done, but a review needs to happen before it can be included in a release.
Release - the goal is waiting to be inckuded in a an official release.
Done - the goal is completed. No further action is necessary.
Blocked - the goal cannot proceed as something is preventing it to move down the pipeline.

For go-ipfs, @whyrusleeping wants a slightly different pipeline:

  ╭─────────────────────────────────── Pipeline ──────────────────────────────────────╮
  │────── 1. ────│──── 2. ───│─── 3. ──│────── 4. ─────│─── 5. ───│──── 6. ───│── 7. ─│
  │ Stages                                                                            │

                                               ╭────────────╮
     ──> Inbox ──> Refining ──> Backlog ──> Ready ──> In Progress ──> Done ──>
                    │             │            │            │ 
                    ╰─────────────╯            │            │
                                            Blocked ────────╯
    Phases
  │────────────── Backlog ─────────────│────── In Progress ───────│──── Complete ─────│
  ╰───────────────────────────────────────────────────────────────────────────────────╯

This re-uses most of the stages but makes these additions/changes:

• Future Work is not used
• Inbox - the goal has not been reviewed or categorized yet
• Refining - the goal has been reviewed but is not ready to be worked on (ie. the description needs clarification, etc.) This is similar to the Future Work stage in the previous diagram.

Important: In both of these diagrams the "Backlog" stage means the same thing. Is that the point of consistency that we're aiming for?

An attempt at Concise Statement

Is this an accurate representation of the consistency we're trying to achieve?

All or our projects have pipeline stages for Backlog, Ready, In Progress and Done. All of those stages will have the same meaning across the projects. Some projects will have additional stages in their pipeline.

Alternatively, are we just saying:

All of our project pipelines distinguish between the three phases of Backlog, In Progress and Done

What else are we trying to communicate?

Background - References to Past Comments on this Topic

These comments talk about this topic at length and include one definition of Backlog

I feel "Backlog" should be a term for the place where all goals are collected in.

The comments seem to converge on some recommended names for sections within the Backlog called either inbox and ready or TODO and Ready

[flyingzumwalt]

@haadcode this ticket is ready for comments now. Based on your feedback on this I think I'll be able to rewire the Backlog and Pipeline section to be really clear.

[flyingzumwalt]

FYI we updated the go-ipfs waffle board to show the pipeline described above.

[jbenet]

Quick comments:

• I would encourage to try to converge on the same names across projects. not a priority now, but over time.
  • eg, i dont care much on "future work" vs "refining" but it would be ideal not to have two words for almost the same concept.
• i would like to be very clear on "backlog" and use the word to mean only one thing. i think it will create confusion if it really means two things. if we mean "the big phase before in progress", then let's not use it as a specific name underneath. (same for "in progress", both show up twice in the ascii diagram for go-ipfs).
• i'm totally ok experimenting with this, if we'll aim to distinguish the words.

[jbenet]

I think this is correct:

All of our project pipelines distinguish between the three phases of Backlog, In Progress and Done

• in that "ready" is part of the "backlog" phase.
• or is "ready" part of "in progress"? from what i've read up to now, i would think backlog..

[haadcode]

Great discussion! Assigning myself so that I remember to reply to this on Monday. WIll get back to you soon :) Quick thoughts: I think we should get rid of the "phases" part in the diagram, it's confusing. "Backlog" == everything before "in progress", "ready", "inbox" etc. being sections within the backlog. Backlog should not be a "stage", but one-level higher or its own concept.

[haadcode]

Also, go-ipfs waffle board looking great! 👍

[serapath]

I'm very curious about ipfs/pm, because maybe other projects can copy and start using the same conventions. I don't know if it's helpful, but will people be paid for work too sometimes? Can tasks be outsourced?

I have some experience using a pipeline to collaborate with other self employed people to get work done.
After quite some projects, we currently use:

1. backlog - to brainstorm/plan. Things here are not yet ready to be implemented
2. ready - which means, this stuff can be done by anyone for money. We pay implementers 50% upfront
3. in progress - it's being worked on
4. review/(invoice) - result is being reviewed by client/product owner and if everything is ok, remaining 50% are being paid
5. done/archrive - items go here when they are finished or canceled for other reasons.

We were thinking of how to make boards composable, thus, if there should be a difference between in progress and some waiting for when tasks are outsourced and need regular followups in case there is no response or a "deadline" is not reached...

Something like "payments" or reached/not reached deadlines trigger transition from one phase of the pipeline to another.

[jbenet]

@serapath

(In short, i think you should not line up your labels to ours, as yours mean something different and they wont directly translate. you want a separate pipeline (one for goal progress and one for contract progress), i think. thankfully most systems allow assigning multiple labels, so you can have multiple boards going on 😄 )

I am curious to learn more about what you guys are doing, the magnitude of your projects, the speed and reliability of development, and so on. Could you point me to a conversation elsewhere, or start one in https://github.com/ipfs/notes or even https://github.com/jbenet/random-ideas ? There is much to say about the cost structures of innovation, how to develop software, and how to develop open source software. On our part, as parties involved in the development of IPFS grow (in capacity and number), there will be more opportunities for helping offset the costs of development (contracts, bounties, and so on -- both to organizations and individuals). But those conversations are very complex, separate to this issue and repo, and should happen elsewhere. So though i'm interested in the conversation, let's discuss elsewhere and let's please keep tickets in this repo on topic. 😄

[flyingzumwalt]

Hi @serapath. I'm glad to hear that you're interested in re-using the model once it's stable. We are certainly thinking of it as a protocol that could be re-used and adapted across many software projects.

In answer to your question about payments for work, remember that these IPFS-related projects are all free, open source projects that are supported by unpaid community contributions. Those contributions are the lifeblood of any healthy open source project.

I want to be clear that marking a ticket as ready in these repositories is not a a request for contract work. Likewise, accepting pull requests into the repositories or resolving github issues is not an agreement to pay for that work.

Companies and organizations who use open source software often contribute to projects by paying their employees or contractors to create code and submit Pull Requests, but those employment arrangements are private to those organizations. They are not part of the public open source project. Some of us have been thinking about these issues at a more systemic level, particularly in terms of funding long term innovation and maintenance of public infrastructure, but as @jbenet said, those are quite complex, so let's keep those elsewhere.

[serapath]

ok. just tried to check where to open and how to name the issue.
Maybe jbenet/random-ideas#21 is a good place? It seems to touch the topic from a different perspective?

[flyingzumwalt]

I think PR #183 addresses this issue enough to close it. @haadcode please speak up if you want to keep the discussion open instead of closing it when the PR is merged.

[serapath]

It seems to be a long document. any sections i should pay special attention to? does it actually consider payment or what exactly do you mean with "addresses this issue enough"?

[flyingzumwalt]

This issue is about providing a clear definition of "Backlog" with guidelines for using it. If people feel that the version of the document in that PR does not achieve that goal, please speak up so we can keep this issue open.

For example, we might want to address #188 and #189 before closing this issue.

@serapath please raise the topic about payment in a separate issue, preferrably in one of the repositories @jbenet suggested. Our current definition of Backlog, Ready, Done, and any other Pipeline status does not have any implications for payment or requests for contract work. If you want to adopt our PM process in your own work and incorporate notions of payment & requests for work, we are curious to see what you come up with but, as @jbenet said, those conversations are very complex, separate to this issue and repo, and should happen elsewhere.

[flyingzumwalt]

The main section of the draft document that deals with "Backlog" is the Workflow section