GitHub Action Commit Queue #2201
Comments
|
I'm in (and I agree with the approach described). Maybe we should fork nodejs/node to work on it (since we'll have to test landing things)? |
|
|
My suggestion is we start by making an action that lands the PR (without any checks at first), because there are restrictions for PRs coming from forks, and we'll need to work around those restrictions. I can take a stab at drafting something tomorrow or Thursday. |
|
Here is one that runs off of the PR being labeled https://github.com/pascalgn/automerge-action |
|
@nschonni thanks, that's probably a good place to start. It'll probably not fit us though since we add metadata to every commit, and usually use pascalgn/automerge-action suffers the fork issues I mentioned above (pascalgn/automerge-action#46). More specifically, for an action make any changes (merge, commit, comment, add labels, etc.) to a Pull Request, it makes a call to the GitHub API using a
Looking at the other events available, I don't see any we could use directly, but here are some ideas:
I think 2 and 3 would have less maintenance burden, and would be easier to ensure only collaborators can land things. 3 could cause a lot of confusion though, so maybe we should try something with the scheduler? Or does anyone have other suggestions? |
|
I'd be generally +1 on this but would definitely like to see it well tested in a separate repo (not core) and accepted before we enable it here. My chief concern is around commit squashing. Most of the time squashing is straightforward, but there are often fixups that we need to do to fix formatting, subsystem in the title, typos, munging of comments from squashed commits, etc, that often really does need to be done manually. |
+1
We can start forcing only single-commit PRs only
I don't think a commit queue is supposed to eliminate manual landing, but rather make it easier to land common cases. It would be the collaborator's discretion to decide if they want to merge via commit queue or manually. |
Initially yes, but as I understand it, the whole idea behind the commit queue is to remove the collaborator's discretion from the mix as much as possible. I'm definitely not against it; but while at times I start to feel like the Old Man Who Hates Everything New, I'd like to see us take our time to get it right. |
|
I think I like the scheduler idea, it fits well with our "wait until land" policy (for example, the second collaborator who approved a PR could add the |
|
I think it's good to have unconditional squashing in the beginning - it seems understandable to me that if you want something more complicated like landing multiple commits with fixups, you'd need to count on a human being to do the chores instead of expecting too much out of automation. If we only ever support single-commit landing in the automation, we also encourage the culture of sending different PRs for different commits if possible - but as always we could leave room for human beings to land things as they wish, instead of making it impossible to land manually. Regarding commit message for the squashed commit - how about taking the PR message? Then the workflow would be somewhat similar to how chromium's commit queue works - before you commit, you change the only CL description which is eventually used as commit message. |
I think 3 sounds safer - it also sounds oddly like the LKGR + master strategy that chromium does, where they have sheriffs to keep an eye on the breakage and revert ASAP without caring too much about how significant the PR is. If we could also adopt a similar culture (prioritize revert + reland over waiting for a fix) that would be a better choice IMO. |
|
I've been playing with the My biggest concern with 3 is:
To elaborate on that, if we have two branches (let's say
Also, automating Revert can get quite complex. (edit: I just double-checked, there's no way to select different branches for the front page and for default PR target) |
Why would this be an issue if the "intend to land" branch is indeed master? It still sends the message that this is not intended to land on e.g. a backport branch. Provided that we still leaves room for people to land commits manually, it still makes sense to point to the master branch and the temporary branch is more like an implementation detail of the action.
Can the action just posts something to IRC and ask a human being to do it ASAP? |
A collaborator (or the author) would still need to change the branch manually. It works, but it's not the most ergonomic experience.
It can also comment on the Pull Request and reopen it I guess. In theory it could just push a Revert commit of itself to I don't think any of the three suggestions are perfect (far from it, as they all are workarounds), so I'm totally fine trying this one out. |
Can't we just keep it as an implementation detail? i.e. the action changes the branch accordingly under the hood and resets master to the the working branch periodically, to people there is just a delay (for verification) in how long the commit eventually lands on master. As a result, nobody can touch master (unless exceptions are made), even people who land the commit manually to the working branch - they will have to wait for the action to sync their landed changes to master (so technically, master becomes LKGR while that working branch is what we currently use as master) |
Is it talking about people doing rebasing/removing, or the action? I think to prioritize the branching strategy for automation we might end up with something like what's done in chromium - the master branch is strictly linear and people can't rebase or remove anything, they could only fix it up by piling revert/reland onto the branch, or just let it go if it's just some minor issues in the commit message, etc. Which seems OK to me |
But the action will only be triggered after merge, in which case the commit will already be on master. Maybe we're thinking different approaches? |
|
@mmarchini I was thinking about something like this commit Meanwhile, another action would also periodically reset the master branch to the last verified commit of |
|
hmm, or maybe the action should trigger a CI (with the rebase option on) before it lands the PR and only actually perform the landing on |
|
Ok, I understand your approach now, and I agree this would be great. The problem is:
The collaborator needs to land it on
When the action runs (with appropriate permissions), the PR is already closed. The whole problem here is that GitHub Actions triggered by events in a Pull Request coming from a forked process do not have edit permission, so it can't change the base branch, close a PR, comment on a PR, or push to master. I also think it can't access secrets (which would be needed to trigger a Jenkins CI), but I'm not sure. So either we still need to use |
|
I see, if the action triggered by the merge can only act on the commits post-merge, it seems something out-of-band (e.g. schedule event) is indeed necessary for formatting the patches. If nothing can be done before the commit lands on a branch (master or not), then we either have to edit one master branch post-merge frequently, or having a unformatted working branch and a formatted LKGR branch out of sync. IMO the first sounds more error-prone to me while it seems possible to work around the second if we maintain that a) the commits on the working branch and the patches on the master branch are in 1-1 correspondence b) the commits on the LKGR branch are formatted versions of the commits in the working branch; c) commits are formatted only once and only done right before they are moved from the working branch to the LKGR branch |
|
If we're fine with an hybrid solution, we could use the |
|
I've been thinking about this again recently. If we want to keep it simple, can we try this flow:
This workflow has two potential issues: 1) if we use the GITHUB_TOKEN to commit those changes, our We'd also need to ensure the If no one objects on this approach I can get something together in the next few weeks (maybe sooner). We can improve later, but I think we need to roll out something. Also, I know we have https://github.com/nodejs/node-auto-test, but maybe we should have a separate repository to try this out? |
|
My other concern would be the spam of revert PRs in case there is a particular incompatibility between a few of them (i.e we land 4 PRs, 2nd changes something that due to changes in 1st PR is now broken and the bot will then open 3 reverts because each CI after 2nd now fails) or which is way more common CI failures due to flaky tests. |
|
@mmarchini I assume that |
PRs are easy to close without merging in case of false-positives, and having PRs will caught the attention of more folks than just a couple of people involved in the original PR. At first I thought about opening a PR only for the first failing CI, but if two subsequent commits fail (I know it's unlikely, but it could happen), the second one would get shadowed. If the notifications are a big concern though, we could just do nothing if the CI fails on master as it will be caught up by someone eventually running CI on a PR, but I worry this would end up confusing users trying to open a PR and having their tests fail for an unrelated reason (and they would probably waste time trying to figure out why it's happening). I'd rather we do something though.
We could. In terms of mention notifications, that would generate the same (or more) than opening a PR. The tradeoff between this and opening a PR is that I would expect the PR to fix master faster because...
...the Actions running on our PRs don't have write permission, thus they can't create a revert as a reaction to a comment. We would need to implement it as another scheduled action (well, it could be the same, just running reverts before landing anything new) and would likely use a Nothing in the proposal is set in stone, so we could go either way (create PR, create issue, comment on PR, do nothing, something else). This can be implemented last and we could try a few approaches. Or we could choose one for now but have the option to change it later. Either way I don't want this to block, and as I said, let's try to keep it simple so we can get something running, even if the solution is not perfect.
Initially yes, but we could also make the Action check if the last full CI finished and was green/yellow, and skip the PR if the CI is still running (this way we wouldn't need to worry about going back to a PR, check if CI finished, and then add a label, which is quite the turnaround consindering our CI takes 40+ minutes). Note that for the approach described in my last comment to work, the Action has to run all the checks we already have in place with node-core-utils, and will only land if all those checks pass (which is also how |
|
Ok, I got the ultra basic workflow working (just land, reverting on failures still tbd)! First PR merged with a commit queue as proposed above on a test repository: mmarchini-oss/automated-merge-test#12. Also, it leaves a comment if This was a lot more tricky than I expected: ncu requires a Jenkins token to read if CI was successful or not, and for some reason the GITHUB_TOKEN doesn't work on ncu when reading some pull requests via the GraphQL API (not all, I assume it's related to who's the PR author). Funny enough, GITHUB_TOKEN works just fine on the scheduler event via the REST API with curl (including write actions), even to the PRs where GraphQL API calls failed with FORBIDDEN. I had to create a personal GH token to circumvent the GraphQL issue (we could create one for I'll document how everything works in the README of the repository above once I get a change, just wanted to share because I'm happy this is finally working ^_^. For anyone curious, most of the logic is in land-stuff.sh. |
|
Ok, I've been thinking more about how to revert when things fail. One option would be to revert in the push workflows, but this has many problems: GitHub doesn't let users declare dependencies across workflows, so either we would need to unify all files which run on push into one, or we would need to duplicate the revert logic on every workflow. Furthermore, GitHub hard stops a job when a step fails, to run a "cleanup step" we would need to add a lot of workarounds to every workflow, regardless of them being in the same file or not. An alternative would be to use another scheduled action, which will look for the last X commits on master and will propose revert for the first one that fails if all subsequent also fails. The upside is that this can reduce noise, but as mentioned before, it could potentially shadow failures on other commits. We could also revert every red commit which doesn't have a revert PR yet. So I'm starting to lean towards manual reverts using the same workflow proposed for landing: collaborators add a 'revert-queue' label to the PR, and the scheduled action does the rest. We might need a |
|
Ugh, just hit another wall and honestly I'm starting to get tired of GitHub Actions 😅 So according to the docs, events triggered by GITHUB_TOKEN will not trigger Action runs. I tested it and confirmed that's the case. The docs also say that if you use a personal token then the Action will run. I tried using a personal token to push but the push event is still not triggered, which means the CI is not running for commits landing on master. I'm not sure if this is a bug on GitHub or if I'm still using GITHUB_TOKEN without knowing. Any help is appreciated, the code is in this repo and I described the implementation in the README. |
|
Ok, never mind, this was an implicit behavior of the checkout action. I needed to pass a personal token to the checkout action too. Will probably start a PR on core tomorrow to discuss adding this. In the first iteration I suggest we leave for collaborators to manually revert: using the commit queue shouldn't be so different from git node land that it would cause more failures on master. In fact, there's not much difference on behavior between the commit queue and a collaborator who decides to bulk land a few PRs at once (the risk of failing on master is the same). Implementing an easier way to revert PRs could be done in parallel to the commit queue initiative. |
This is a (still experimental) implementation of a Commit Queue on GitHub Actions, using labels and the scheduler event to land Pull Requests. It uses `node-core-utils` to validate Pull Requests and to prepare the commit message, and then it uses a GitHub personal token to push changes back to the repository. If the Queue fails to land a Pull Request, that PR will be removed from the queue and the `node-core-utils` output will be pasted in the Pull Request. An overview of the implementation is provided in doc/guides/commit-queue.md, as well as current limitations. Ref: https://github.com/mmarchini-oss/automated-merge-test Ref: nodejs/build#2201
This is a (still experimental) implementation of a Commit Queue on GitHub Actions, using labels and the scheduler event to land Pull Requests. It uses `node-core-utils` to validate Pull Requests and to prepare the commit message, and then it uses a GitHub personal token to push changes back to the repository. If the Queue fails to land a Pull Request, that PR will be removed from the queue and the `node-core-utils` output will be pasted in the Pull Request. An overview of the implementation is provided in doc/guides/commit-queue.md, as well as current limitations. Ref: https://github.com/mmarchini-oss/automated-merge-test Ref: nodejs/build#2201 PR-URL: #34112 Refs: https://github.com/mmarchini-oss/automated-merge-test Refs: nodejs/build#2201 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Shelley Vohr <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Richard Lau <[email protected]>
This is a (still experimental) implementation of a Commit Queue on GitHub Actions, using labels and the scheduler event to land Pull Requests. It uses `node-core-utils` to validate Pull Requests and to prepare the commit message, and then it uses a GitHub personal token to push changes back to the repository. If the Queue fails to land a Pull Request, that PR will be removed from the queue and the `node-core-utils` output will be pasted in the Pull Request. An overview of the implementation is provided in doc/guides/commit-queue.md, as well as current limitations. Ref: https://github.com/mmarchini-oss/automated-merge-test Ref: nodejs/build#2201 PR-URL: #34112 Refs: https://github.com/mmarchini-oss/automated-merge-test Refs: nodejs/build#2201 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Shelley Vohr <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Richard Lau <[email protected]>
This is a (still experimental) implementation of a Commit Queue on GitHub Actions, using labels and the scheduler event to land Pull Requests. It uses `node-core-utils` to validate Pull Requests and to prepare the commit message, and then it uses a GitHub personal token to push changes back to the repository. If the Queue fails to land a Pull Request, that PR will be removed from the queue and the `node-core-utils` output will be pasted in the Pull Request. An overview of the implementation is provided in doc/guides/commit-queue.md, as well as current limitations. Ref: https://github.com/mmarchini-oss/automated-merge-test Ref: nodejs/build#2201 PR-URL: #34112 Refs: https://github.com/mmarchini-oss/automated-merge-test Refs: nodejs/build#2201 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Shelley Vohr <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Richard Lau <[email protected]>
This is a (still experimental) implementation of a Commit Queue on GitHub Actions, using labels and the scheduler event to land Pull Requests. It uses `node-core-utils` to validate Pull Requests and to prepare the commit message, and then it uses a GitHub personal token to push changes back to the repository. If the Queue fails to land a Pull Request, that PR will be removed from the queue and the `node-core-utils` output will be pasted in the Pull Request. An overview of the implementation is provided in doc/guides/commit-queue.md, as well as current limitations. Ref: https://github.com/mmarchini-oss/automated-merge-test Ref: nodejs/build#2201 PR-URL: #34112 Refs: https://github.com/mmarchini-oss/automated-merge-test Refs: nodejs/build#2201 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Shelley Vohr <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Richard Lau <[email protected]>
This is a (still experimental) implementation of a Commit Queue on GitHub Actions, using labels and the scheduler event to land Pull Requests. It uses `node-core-utils` to validate Pull Requests and to prepare the commit message, and then it uses a GitHub personal token to push changes back to the repository. If the Queue fails to land a Pull Request, that PR will be removed from the queue and the `node-core-utils` output will be pasted in the Pull Request. An overview of the implementation is provided in doc/guides/commit-queue.md, as well as current limitations. Ref: https://github.com/mmarchini-oss/automated-merge-test Ref: nodejs/build#2201 PR-URL: #34112 Refs: https://github.com/mmarchini-oss/automated-merge-test Refs: nodejs/build#2201 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Denys Otrishko <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Shelley Vohr <[email protected]> Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: Richard Lau <[email protected]>
|
This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made. |
My thinking is let's focus on making this a GitHub Action that can be run when Jenkins is green/yellow. Maybe triggered by a Collaborator comment? It can re-run some things as GitHub Actions to make sure tests still pass on the most popular platforms before adding metadata and merging.
Who wants to be involved? We should probably add and remove people from @nodejs/commit-queue as appropriate.
Refs: nodejs/commit-queue#1
The text was updated successfully, but these errors were encountered: