Our proposal for making contributing easier

[ErikSchierboom]

Over the last couple of months, we've had tons of great contributions to v3. While reviewing and merging this PR's, we found that one issue that kept popping up was that it could take a long time for PR's to be merged. This was both frustrating for the reviewer and the contributor. There are several reasons for PR's being open a long time:

• Exercises are comprised of many files: documentation files, implementation files and updates to metadata. Many files automatically mean more reviewing time and a bigger chance of comments.
• Some PR’s would be submitted as a "fully fleshed-out PR’s" but had quite fundamental mistakes, which meant that the PR had to be changed quite a bit.
• Too many people doing reviews. After fixing one reviewer's issues, the contributor would sometimes find that a new reviewer requests additional changes.
• Too much focus on getting things 100%. The best example here is grammar and spelling. For many of our contributors, English is not their native tongue, which means that there would sometimes be dozens of grammar and spelling comments.

Having established that there are several reasons why PR reviews could take a long time, how can we fix this? The most obvious solution is to somehow get PR's to be smaller. Here are the solutions that we've come up with:

• Use an "Optimistic Merging" approach to PR's, which means if it’s "good enough" we’ll merge it and improve it later. That means PR submitters don’t need to worry about getting everything perfect first time and reviewing can be more rewarding. You can read more about optimistic merging and its benefits here.
• Require only three files for a new exercise: the design.md, instructions.md and an example implementation file. With just these three files, a reviewer should be able to graps if the exercise's design is sound. Having just three files will mean less churn and a (far) lower barrier to contributing.
• Use draft PR's for new exercises, which signals to the reviewer and contributor that the PR is meant to be a starting point for a high-level discussion on the exercise.
• Create a web-based wizard to allow contributors to easily put together a draft PR for a new exercise. The contributor will be able to fill in a form and the end result will be a draft PR. This will lower the barrier for contributors, as not all contributors will be familiar with Git and GitHub.
• We'll automatically post a comment on PR's that add a new Concept Exercise. This comment will contain useful information for both the contributor and reviewer. It will also include a nice and convenient checklist, which will indicate the status of the PR.
• As we'll be merging PR's sooner, most of the times not all files will have been added or updated for an exercise. To still be able to keep track of the status of an exercise (which files to add or update in follow-up PR's), we'll add readiness fields to the .meta/config.json file of an exercise:

{
  ...
  "readiness": {
    "docs": {
      "after": false,
      "hints": false,
      "introduction": false,
      "instructions": true
    },
    "exercise": {
      "example": true,
      "stub": false,
      "tests": false
    },
    "meta": {
      "config-json-for-exercise": false,
      "config-json-for-track": false,
      "design": true,
      "english-language-checked": false,
      "reference-doc": false,
      "story": false
    },
    "misc": {
    }
  }
}

Additional keys can be added "misc" key can be used for track-specific checks.

• We'll add functionality to automatically convert a draft PR to an actual PR (using a GitHub Actions workflow). In this step, we'll try to automatically add files based on what we know about the PR. For example, we can automatically create (part of) the .meta/config.json file by looking at the draft PR author and the checkboxes set in the checklist that is part of the PR's automatic comment.

With the above changes, we're hoping that the PR process will be streamlined significantly, making it far more fun for both the reviewer and the contributor. The process should become more iterative, with smaller steps. This has the additional benefit that it becomes easier to contribute. People can also focus on the parts where they feel they can make the best contribution; if you don’t want to build an exercise but you love reviewing grammar and spelling, please do!

---

A lot of this thinking has come from discussions with maintainers and from the weekly calls. We hope this puts together all those ideas into a sensible system. Before we implement this everywhere, have we missed anything? Do we have any blind-spots? Is there anything we could change further to make this even better?

[chocopowwwa]

I was thinking about this too while listening to the last meeting.

I think tracking each files is a correct way, but if you track the status in .meta/config.json that would've been hard if you want to check the overall status of the project, I'd suggest we track the status in a Google sheet, where each page represents a track, and reach row represent the concept and each column represent the status

The status can be:

• none
• need improvement
• in progress (with hyperlink to the PR)
• done

What do you think?

[chocopowwwa]

Oh, and also I think for the initial file, from none to need improvement, I think we need to have some kind of prerequisite, for instance It wouldn't make sense to have an example without having the instructions first

Here's my suggestion:

• after.md
• instructions.md
• introduction.md
• instructions.md -> hints.md
• instructions.md -> tests
• instructions.md -> example
• instructions.md -> example -> stub

[neenjaw]

I usually write the example solution before the instructions. I find it easier to determine if the solution is to big/small/complex/simple etc.

[chocopowwwa]

I'm not suggesting the process, I'm suggesting on the set of files which the commit has to have. For instance, if you're going to write your example first, it's fine, but make sure when you commit your example you have your instruction in place because I think they have some sort of interdependency, I think if you commit the example then someone commits the instructions, the instructions might be different from your example, you might have to re-do the example, same thing the stub, tests, and hints.

[tehsphinx]

While I understand, that the instructions have an effect on the example (especially the story part), I found that people inventing the story first come up with way to complex exercises. That is why I prefer to write the example first, ideally coming up with a rough story at the same time in my head. So for me: definitely example first.

[ErikSchierboom]

I think this is highly personal and that we should not attempt to add an ordering. I also write the example first, and then the instructions after that.

[tehsphinx]

that would've been hard if you want to check the overall status of the project

Should be quite easy if it gets implemented in https://tracks.exercism.io/. And I'd rather have it in the repository than in another place like a Google Sheet that will be out of sync in no time.

[SleeplessByte]

This could definitely be implemented on the dashboard (cc @tehsphinx).

I think we need to have some kind of prerequisite, for instance It wouldn't make sense to have an example without having the instructions first
#1482 (comment)

Already happened. #1390

[neenjaw]

I also don't think it serves much purpose to force an order for the completion of the files. I also write the example solution before the instructions.

[SleeplessByte]

I don't think adding more JSON to the config is the best solution. I understand the implementation. I do think we should do this track maintenance dashboard style. If the JSON would be changed automagically anyway, it's just more noise to the config file with potential issues.

@ErikSchierboom I like all the other suggestions a lot; just the implementation is what I comment on here 🙉

[ErikSchierboom]

Thanks! I understand what people don't like about it. Let's think on it for a bit!

[ghost]

Thank you for taking the time to think through this and propose better approaches. I will provide better feedback when I free up more.
I would love for us to use the "resolve conversation" feature in pull requests. This helps me as a time-constrained maintainer focus on new / outstanding review items.
I think we need to do 2 things

1. determine who should be responsible for resolving conversations
2. document this suggestion somewhere

[ErikSchierboom]

I can totally see this resolving thing go both ways. The most fool-proof way would be for the person that added the comment to resolve (as the maintainer could possibly have not solved in the way the commenter intended), but the most convenient way would be for the maintainer to do so (as that indeeds allows the maintainer to focus). I could be swayed both ways, but I have a slight preference for the maintainer being able to resolve things.

[wolf99]

• Too many people doing reviews. After fixing one reviewer's issues, the contributor would sometimes find that a new reviewer requests additional changes.

Perhaps this could be resolved by using the "Assignee" functionality of GitHub a little more

As an example:

We could ask that reviewers add themselves to the "Assignees" on a PR:

• If there are two already, don't add yourself and don't review.
• If you are an assignee and are no longer able to contribute on a PR, remove yourself from the assignees
• If one or both of the previous assignees have not returned to a PR, add a comment to ping them on the PR
• If a pinged assignee has not been able to respond, change out that assignee, but still discuss/resolve points that reviewer has raised.

This reduces the effect mentioned by Eric, but also means that reviewer effort isn't overly focused in some areas while leaving other areas without reviews.

[ghost]

These are exciting changes! I have felt similar problems creating and reviewing PRs for the rust track.

it could take a long time for PR's to be merged

Curious, what duration do you expect for merging? I imagine this differs for draft versus "final" ones. For me, it would be nice to see a line explaining the expected timelines.

💯 and 👍 for the

• optimistic merging

• 3 file requirement for the concept exercise MVP

• draft PRs for new exercises

• web wizard for contributors unfamiliar with Git and GitHub

• modeling readiness state and automating its creation

  We'll automatically post a comment on PR's that add a new Concept Exercise. This comment will contain useful information for both the contributor and reviewer. It will also include a nice and convenient checklist, which will indicate the status of the PR.

I think this might become stale in the process of multiple review iterations. So I would need to remember to check the list upon each change. But pivoting the main review process to smaller PRs should render this objection mute or less significant.

[ErikSchierboom]

Curious, what duration do you expect for merging? I imagine this differs for draft versus "final" ones. For me, it would be nice to see a line explaining the expected timelines.

Well, what I don't want is for PR's to be open for weeks. Draft PR's should ideally be up for a couple of days at most, while "final" ones could take longer. But getting things merged within a week or so sounds like a reasonable string to strive for, although there could of course be lots of circumstances that have it be up longer. But aiming for at most a week and preferrably days sounds like the best plan.