Automatically create pull requests based on Homebrew/livecheck/Repology

[MikeMcQuaid]

Homebrew/livecheck provides various automated ways of detecting formulae updates.

Repology provides an API that provides details on whether a Homebrew package is outdated e.g. https://repology.org/metapackage/boost/versions

A separate Ruby application that can be trivially deployed on e.g. Heroku should be built to make use of these checks, verify that the results seem correct and open a pull request with relevant metadata on Homebrew/homebrew-core.

No need to say "I'm planning on working on this" or similar in this issue but feel free to open a PR that doesn't work (yet!) and ask for help.

[BenMusch]

Hey Mike! I'm interested in this concept, but it seems like the work that goes into making a separate application (creating a repo, hosting the app, storing secret keys) would be something better suited for a member of the homebrew team to work on. Is that correct, or is it possible for an outside contributor to work on this?

[scpeters]

My first thought when I read this was that it could be a command-line application. For example, brew livecheck already has a command-line interface, and brew bump-formula-pr can create pull requests from the command-line, so I imagine that a solution could work in that same style, which wouldn't require storing secret keys or hosting an app.

[BenMusch]

That makes sense, thanks! I'll look into this and comment back here if I make significant progress on this concept

[MikeMcQuaid]

@scpeters Adding this logic to e.g. brew bump-formula-pr (or another command that wraps/feeds it) would be a good starting point, I agree. Eventually I do think an application that does this without human intervention would be preferable.

@BenMusch Given the above I think you should be able to get started. It's worth noting that even an application wouldn't need to store any secret keys; you could currently write an application to submit these pull requests as yourself as there's no special permissions required to submit PRs. Given e.g. Heroku's free tier it should be pretty easy to get started on a simple server for this in a public repo both of which could be migrated to Homebrew at a later stage.

[jeduardo824]

Could I start to work on this?

[zachauten]

@jeduardo824 Sure! I’ve been looking into it as well, let me know if you have any questions/ideas!

[MikeMcQuaid]

Yes please to either of you!

[zachauten]

@MikeMcQuaid What should the threshold for an upgrade be? I imagine there may be situations where we might not want to automatically bump major versions of a package, and bumping hundreds of packages every time there’s a new patch version could create a lot of noise.

[MikeMcQuaid]

What should the threshold for an upgrade be?

A new version being available.

bumping hundreds of packages every time there’s a new patch version could create a lot of noise.

I'm not sure I understand this, can you elaborate?

[zachauten]

Sorry, never mind that part about patch versions; My thinking was that creating a large amount of PRs at once to update formulae by a small amount would be annoying to manage, but that would only be the first time you run it.

[larson-carter]

Ok, I just jumped into this issue. So you want to automatically open a PR against Homebrew/homebrew-core if it detects a package being out of date?
@MikeMcQuaid

EDIT: typo

[MikeMcQuaid]

@larson-carter I would start with the "a separate application should be built to make use of these checks" and not automatically open any PRs yet.

[larson-carter]

@larson-carter I would start with the "a separate application should be built to make use of these checks" and not automatically open any PRs yet.

Ok, I'll start to work on it!

[larson-carter]

@MikeMcQuaid I have started work on this automated checker. Please see the link attached here with my current progress. Do you have a tutorial on how to update packages manually?

https://github.com/larson-carter/Homebrew-Package-Checker

[MikeMcQuaid]

Nice start @larson-carter and sorry for the delay! I've edited the issue above to note this will need to be written in Ruby for us to maintain, unfortunately. brew bump-formula-pr is the best way to submit package version updates. Thanks!

[larson-carter]

@MikeMcQuaid

Hey no worries about the delay.

I'll re write it in Ruby, then we can probably work something out since it will be nested under a completely new repo also for transferring ownership to the homebrew org. I'll also take into account about > brew bump-formula-pr

[zachauten]

I have a pretty basic solution here: https://github.com/zachauten/bump. It collects formulae that match between repology and livecheck, plus livechecks that aren't guessed, and runs bump-formula-pr on them.

Still lots of edge cases & peculiarities that could be ironed out.

[MikeMcQuaid]

@zachauten Nice! Is that something you're using yourself for version bumps? If not, that would be an interesting starting point.

[zachauten]

@MikeMcQuaid I've used it on one formula at a time, testing. Going to try a "real" run tonight.

[MikeMcQuaid]

@zachauten Nice, let us know how it goes!

[zachauten]

@MikeMcQuaid So the run yesterday picked up 91 outdated formulae that matched between livecheck and repology. About half passed the CI.

[alecclarke]

@MikeMcQuaid I'm not sure if you're still looking for a solution on this one, but I created a proof-of-concept project/repo that can leverage GitHub action's scheduling trigger to update Livecheck formulae automatically.

Currently the code invokes brew bump-formula-pr using the --dry-run flag so as to not make any noise from creating PRs. Also, I'm using a .brew_livecheck_watchlist list to keep the action run time low, but that can easily be replaced with adding --all to the brew livecheck call.

Would this approach be acceptable to what you're looking? If so I'm happy to keep working away on this.

[MikeMcQuaid]

@alecclarke Yeh, we're still looking for a solution. Can you link to some output of your GitHub Action?

[alecclarke]

@MikeMcQuaid sorry for the delay.

Here's a link to the latest Action run. You can see the output under Run brew bump section.

This is running off the formulae listed in .brew_livecheck_watchlist. Some of these have open PRs to bump the version already and that's also shown in the Run brew bump output.

[ran-dall]

@MikeMcQuaid We could probably extend the ML system we use for cask to support something like this. The only thing I see a problem with is that it isn't Ruby-based but more C and R based more than anything else. However, it can do a lot more than just make a PR when there's a new version of something available.

[larson-carter]

@MikeMcQuaid

We could possibly just expand on this project: https://github.com/MLH-Fellowship/0.0.1-API-Parser

All we need to do is have the bot open a PR against the formulae.

[alecclarke]

Thanks, @MikeMcQuaid !

Are there any incremental changes to Homebrew/brew or Homebrew/homebrew-livecheck that you can think of making based on your work so far?

Would it be reasonable to add more details to the json output of Livecheck?

Specifically, if we could add the new formula version's url to the output, or in the cases where a git tag based strategy is used, add the revision + tag - that would remove the need for brew bump to access Formula and be run as an external brew command.

[MikeMcQuaid]

Would it be reasonable to add more details to the json output of Livecheck?

I think so! @samford has been working on similar stuff and they may have some ideas here?

[samford]

I've been improving the debug and JSON output in Homebrew/homebrew-livecheck#860, so you may be interested in that.

In addition to the existing changes in that PR, I'm in the process of adding the following information to the JSON output (to bring it in line with the information that PR presents in the debug output):

• URL
  • The unmodified URL from the formula or livecheckable
  • The processed URL (i.e., the output from preprocess_url), if it differs
  • Any generated URL used in the strategy, if it differs
• Strategy used
• Regex (from the livecheckable or a default regex used in a strategy, if any)

This requires passing information from the strategy methods (in livecheck/livecheck_strategy.rb) back to latest_version() (in livecheck/heuristic.rb) and back to print_latest_version() (in cmd/livecheck.rb). Currently this doesn't happen, so I'm having to modify everything in this chain to make this information available where the JSON is generated in print_latest_version() and provided to livecheck() (which outputs the JSON).

Beyond this, it may be worth mentioning that I'm keeping the default JSON output as minimal as possible and requiring use of another flag in addition to --json to get the lengthier output including the above information. I'm currently using the --verbose flag in the PR for this but there's debate about whether --debug --json would make more sense.

There are also some changes to field names in the aforementioned PR, so if you're currently working with livecheck's JSON output in some way, you'll want to pay attention to these forthcoming changes.

---

If there's any additional information that's available to livecheck and you could benefit from having in the JSON output, just let me know and I can work on including it.

I'm unsure of precisely what's meant by "the new formula version's url". If you mean the URL for the archive file of the new version (e.g., https://example.com/example-1.2.3.tar.gz), this isn't currently possible with livecheck. We only capture the version text like 1.2.3 and not a full archive URL, so we would have to make some fundamental changes and modify hundreds of livecheckables to be able to provide this information. If I've understood this correctly, I can try to iron out how to make it work and it may be something that our GSoC student (Nanda) could work on.

That said, this isn't going to be possible for 100% of formulae, let alone 100% of the formulae for which livecheck currently works. For some of the checks, the URL of the new archive isn't available where we're checking (whereas the version number is). For formulae with very consistent URLs and filenames, you could consider substituting the new version string in the URL for the interim time and checking whether that URL is good (e.g., using a HEAD request). [Some URLs redirect to a mirror, so the criteria for "good" is more than just a 200 response.]

Regarding the Git strategy, it uses git ls-remote --tags and parses the tags from the output. The commit hash is also available in this output, so I could modify the git_tags() function to parse it as well if that's what you mean by "revision".

[alecclarke]

@samford thanks for the details!

I'm unsure of precisely what's meant by "the new formula version's url". If you mean the URL for the archive file of the new version (e.g., https://example.com/example-1.2.3.tar.gz), this isn't currently possible with livecheck.

Yes, that's what I was wondering about. In the brew bump command I've been working on I was using the strategy of simply substituting the new version string in the URL (like you described) but I was hoping that their might be a more robust way of getting that value. If you think there's value identifying the new formula version's URL and adding it to the Livecheck output, I'd be happy to contribute to that effort.

Regarding the Git strategy, it uses git ls-remote --tags and parses the tags from the output. The commit hash is also available in this output, so I could modify the git_tags() function to parse it as well if that's what you mean by "revision".

Yes, that's what I mean by revision. If the Livecheck output included the tag and commit hash that would make bumping formulae with git based downloaded strategies much easier!

While it looks like you are already well underway on making some of these changes, let me know if/where I can help out with this work!

[MikeMcQuaid]

Beyond this, it may be worth mentioning that I'm keeping the default JSON output as minimal as possible and requiring use of another flag in addition to --json to get the lengthier output including the above information. I'm currently using the --verbose flag in the PR for this but there's debate about whether --debug --json would make more sense.

I'd be tempted to make the JSON always be the same unless it's significantly more expensive to generate. If so, --verbose would be preferable to --debug (which has other effects).

While it looks like you are already well underway on making some of these changes, let me know if/where I can help out with this work!

@samford Would be cool to get a WIP PR open at some point and then @alecclarke and others could take a look!

[cnnrmnn]

@MikeMcQuaid Is this issue still pending or was it addressed in #7834?

[samford]

@cnnrmnn #7834 introduced the brew bump command, which checks repology.org, brew livecheck, and open PRs, and displays this information. brew bump doesn't take any action based on what it finds, so this is still an open issue and I believe it warrants further discussion.

At the moment, it's technically possible to use livecheck's data to automatically version bump a subset of formulae that are either simple or consistent. @dawidd6 created a GitHub Action that can be used for this and, outside of the linked examples in the README, we have a homebrew/core maintainer who regularly uses this to create automated version bump PRs (see https://github.com/chenrui333/github-action-test/).

Though this works for some formulae, there are a number of others that are more complicated. I believe part of what needs to be discussed in this issue is which formulae are problematic, what the issues are, and where changes need to be made to support them (e.g., livecheck, the Formula DSL, tooling for creating automated PRs, etc.). Additionally, there's at least one issue that needs to be addressed to improve the signal-to-noise ratio of automated PRs before it becomes a widespread thing (see below).

The most active homebrew/core maintainers likely have the best idea of how hit-or-miss the existing automated version bump PRs have been so far and what kind of issues they've encountered.

---

Off the top of my head, one issue we had with an earlier attempt at this (before Dawid's GitHub Action) was that the tooling didn't account for closed PRs. Some formulae versions don't build/test successfully and we can't update until something changes (upstream, reworking the formula, etc.) but this isn't apparent unless you dig through closed PRs. This is something that humans can easily check but we don't currently have a way of managing this metadata. A closed PR can indicate any number of different things, so a tool would need a stronger signal than that to know when to skip an automated version bump.

This is not only important for tooling but could also act as a signal for contributors/maintainers to avoid wasted effort. There are times where someone will forget to check closed PRs and start working on updating a formula (or use brew bump-formula-pr) only to find there's a roadblock preventing it from working. Having this information available somewhere would help prevent low-value PRs and wasted effort for both people and CI.

How this is handled (e.g., adding something to the Formula DSL?) is something that requires discussion. It may also tie into the general idea of whether automated version bumping should be opt-in or opt-out (though this depends on how intelligent and successful any related tool ends up being in practice).

[Rylan12]

Do you have a sense of how often a PR is closed not because it's unable to be built at the moment?

I can't think of many situations where an automatic version bump PR would solve an issue that a maintainer or contributor was unable to solve manually, so it seems counterproductive to open new PRs at all if there's already been an attempt. Now, I could be missing a key situation where we would want this behavior.

One thing we could do is have a label (e.g. retry) that would indicate that the reason the PR was closed should not invalidate it from future version bumps. For example, maybe a maintainer opened too many PRs and just needed to close some to keep the load manageable.

---

Some other thoughts just to add to the board, here:

It might be worth considering adding a limit to the number of PRs that are automatically opened per hour/day. Or, maybe, the number of automated PRs open at any given time should be capped. I know that we sometimes run into issues where the automatic updates are running faster than we can review depending on maintainer availability.

It also might be helpful to have some sort of "dashboard" where the status of all formulae could be viewed. That way, at a glance, people could see which formulae need manual work and which ones are or will be automatically bumped. I guess it could be extended even more to show whether there are any PRs open that modify a given formula. Maybe this is just a GitHub issue that we lock and is just a list of outdated formulae that is automatically updated every once in a while. This is the kind of thing that would be cool to have but may not be worth the effort if we don't think it would be helpful.

[cnnrmnn]

Thanks for the detailed response, @samford. I read @dawidd6's GitHub Action and the documentation for brew livecheck.

Assuming that the tooling automating version bump PRs generates consistent messages (as bump-formula-pr does), you could query closed PRs to determine if an automated version bump should be skipped. This isn't a particularly dynamic approach, but it would at least prevent duplicate/low-value PRs. If there's some uncommon/formula-specific roadblock preventing an automated version bump, then some human intervention would be required.

If there are common roadblocks that prevent automated version bumps, perhaps something could be written to monitor the status of the roadblock and retry the bump when appropriate. I don't know enough about the most frequent issues.

Do you think something like this belongs in a brew command or in a GitHub Action like Dawid's solution?

[cnnrmnn]

Do you have a sense of how often a PR is closed not because it's unable to be built at the moment?

I can't think of many situations where an automatic version bump PR would solve an issue that a maintainer or contributor was unable to solve manually, so it seems counterproductive to open new PRs at all if there's already been an attempt. Now, I could be missing a key situation where we would want this behavior.

One thing we could do is have a label (e.g. retry) that would indicate that the reason the PR was closed should not invalidate it from future version bumps. For example, maybe a maintainer opened too many PRs and just needed to close some to keep the load manageable.

@Rylan12 Agreed. I don't have a good sense of the frequency of categories of issues with bump PRs, but it seems like automating bump PRs is most valuable for the trivial bumps so that contributors can spend more time on more complicated ones. It would be pretty simple to implement something that doesn't duplicate closed PRs.

It might be worth considering adding a limit to the number of PRs that are automatically opened per hour/day. Or, maybe, the number of automated PRs open at any given time should be capped. I know that we sometimes run into issues where the automatic updates are running faster than we can review depending on maintainer availability.

An alternative here could be to monitor the total number of open PRs and not create any automated PRs once the total meets a threshold.

[Rylan12]

Assuming that the tooling automating version bump PRs generates consistent messages (as bump-formula-pr does), you could query closed PRs to determine if an automated version bump should be skipped.

@Rylan12 Agreed. I don't have a good sense of the frequency of categories of issues with bump PRs, but it seems like automating bump PRs is most valuable for the trivial bumps so that contributors can spend more time on more complicated ones. It would be pretty simple to implement something that doesn't duplicate closed PRs.

Assuming such a command would use brew bump-formula-pr to create the automatic PRs, this would be taken care of. bump-formula-pr already checks for PRs with matching names and won't create a new one unless the --force flag is passed (which I wouldn't expect to happen for automated version bumps).

I think the bigger issue is how to know which formulae with which we do want to try again (although maybe I misunderstand the problem that @samford is describing).

Do you think something like this belongs in a brew command or in a GitHub Action like Dawid's solution?

I think it depends on what the specific solution is. We already have a bunch of the parts working (i.e. brew livecheck, brew bump, brew bump-formula-pr). I think all that's missing is the logic linking them to each other. I wouldn't be opposed to a brew command that handles all of it. That way, this could even be extended to third-party taps pretty easily. Plus, writing it in Ruby in the brew repo has a lot of advantages and I think is much easier to maintain.

The reason I could see choosing a GitHub action over a brew command would be if the brew command is so simple that it's literally just executing other brew commands. In that case, it would be great to get it as an action in the Homebrew/actions repo.

[cnnrmnn]

Assuming such a command would use brew bump-formula-pr to create the automatic PRs, this would be taken care of. bump-formula-pr already checks for PRs with matching names and won't create a new one unless the --force flag is passed (which I wouldn't expect to happen for automated version bumps).

Didn't realize this. Thanks for letting me know.

I think the bigger issue is how to know which formulae with which we do want to try again (although maybe I misunderstand the problem that @samford is describing).

Even if the initial solution doesn't retry bumps for any formulae with closed PRs, I think that it would be an improvement over the GitHub Action for the reasons you mentioned.

I can start on a solution that pieces the existing parts together into a brew command. If it turns out that there are common situations where it makes sense to retry bumps, I can extend what I write to handle them. Thoughts?

[nandahkrishna]

I wouldn't be opposed to a brew command that handles all of it.

Just adding a small note here that brew bump was supposed to be the command linking livecheck and bump-*-pr. It already does get the necessary data from livecheck and GitHub (open PRs, could be extended to other kinds of PRs too), all it needs to do is run bump-formula-pr or bump-cask-pr.

I'd been working on the initial steps towards this, but if anyone else would like to open a PR, please feel free.

I think the bigger issue is how to know which formulae with which we do want to try again

Do you have a sense of how often a PR is closed not because it's unable to be built at the moment?

We see a few PRs closed in homebrew/core because the software is a pre-release (it isn't the latest yet on GitHub Releases, upstream explicitly mentions that it's a pre-release, etc). Mostly, the PR is left open for a few days until some solution is found.

Build failure PRs would be closed if nobody was able to work on fixing the build failures or if a new release was required (rare) for things to work. Pre-release PRs would be closed depending on whether there's a response from upstream (or discussion in an upstream issue), whether the livecheck block has to be tweaked or added, etc.

[Rylan12]

Just adding a small note here that brew bump was supposed to be the command linking livecheck and bump-*-pr. It already does get the necessary data from livecheck and GitHub (open PRs, could be extended to other kinds of PRs too), all it needs to do is run bump-formula-pr or bump-cask-pr.

Yeah, I'd say just add on to brew bump in that case

Pre-release PRs would be closed depending on whether there's a response from upstream (or discussion in an upstream issue), whether the livecheck block has to be tweaked or added, etc.

Assuming the live check isn't updated, we have no way to determine whether it's still a prerelease or not so it shouldn't be opening a new PR anyway. I guess there might be a challenge for cases where the livecheck is updated and so the next time a version update is detected a PR should be opened even though one already existed. I think in these cases the best thing to do may be just to rename the PR so it's clear that it's not really a version bump. Alternatively, we can probably filter out "merged" PRs (as opposed to "closed" PRs) when checking for duplicates.

[MikeMcQuaid]

I can't think of many situations where an automatic version bump PR would solve an issue that a maintainer or contributor was unable to solve manually, so it seems counterproductive to open new PRs at all if there's already been an attempt. Now, I could be missing a key situation where we would want this behavior.

This is the main issue I see with the existing automation (CC @chenrui333). I think we should start thinking about how to make brew bump or a similar GitHub Action actually attempt an installation (or, ideally, full brew test-bot run) before it's opening a PR. This doesn't need the same level of OS coverage we provide or e.g. testing all dependents but it would be good to check bump PRs are building before a PR is opened rather than after (or: we consider autoclosing PRs that fail CI and are autoopened).

Once we have linuxbrew-core and homebrew-core merged (and a lot more Linux bottles) this could be easier (as Linux compute is much more readily and cheaply available).

Just adding a small note here that brew bump was supposed to be the command linking livecheck and bump-*-pr. It already does get the necessary data from livecheck and GitHub (open PRs, could be extended to other kinds of PRs too), all it needs to do is run bump-formula-pr or bump-cask-pr.

Yeah, I'd say just add on to brew bump in that case

Agreed.

[samford]

I can't think of many situations where an automatic version bump PR would solve an issue that a maintainer or contributor was unable to solve manually, so it seems counterproductive to open new PRs at all if there's already been an attempt.

Livecheck isn't infallible, so there may be uncommon exceptions. However, I think it's generally reasonable to assume a closed PR means we shouldn't create an automated version bump PR for a given formula/version combination.

I agree it would be good to have a label that we can apply to prevent a PR from being used to exclude a formula/version combination. I don't imagine it would be used too often but we'll need it for some situations (e.g., a PR is created for a prerelease version and closed, we fix the livecheck block to avoid prerelease versions but we want to bump to the version once livecheck reports it as stable). This strikes me as easier than having to decide what to rename a failed PR to (or having to remember whatever renaming convention we would decide on).

---

It also might be helpful to have some sort of "dashboard" where the status of all formulae could be viewed.

You've anticipated me, ahah. This is something that I've been quietly thinking about since late 2019 but I've only hinted at it a few times. I believe there's value in making adverse status information available in a standardized way (e.g., a formula DSL, a JSON file in the tap, etc.) as well as aggregating related information in one place (e.g., a dashboard).

However, I don't think the former would currently impact a large number of formulae from the standpoint of automated version bumps, so I'll create a separate issue to outline my ideas around this. There may be some parts that are more important in the near-term (and we can prioritize those) but I don't want to make this issue unmanageable by adding a parallel discussion on top.

---

I think we should start thinking about how to make brew bump or a similar GitHub Action actually attempt an installation (or, ideally, full brew test-bot run) before it's opening a PR.

I tend to agree with this. The homebrew/core PR template makes it clear that we want people to build/test a formula locally before opening a PR but brew bump-formula-pr has often been [ab]used in a "let CI build/test it so I don't have to" fashion. If it's not part of the tool/action's normal behavior, some people aren't going to bother.

[MikeMcQuaid]

I agree it would be good to have a label that we can apply to prevent a PR from being used to exclude a formula/version combination. I don't imagine it would be used too often but we'll need it for some situations (e.g., a PR is created for a prerelease version and closed, we fix the livecheck block to avoid prerelease versions but we want to bump to the version once livecheck reports it as stable).

I like the label idea here, makes a lot of sense to me 👍🏻

You've anticipated me, ahah. This is something that I've been quietly thinking about since late 2019 but I've only hinted at it a few times. I believe there's value in making adverse status information available in a standardized way (e.g., a formula DSL, a JSON file in the tap, etc.) as well as aggregating related information in one place (e.g., a dashboard).

This could be a GitHub Actions job that's run nightly and commits a (JSON?) file to formulae.brew.sh to be displayed there.

I tend to agree with this. The homebrew/core PR template makes it clear that we want people to build/test a formula locally before opening a PR but brew bump-formula-pr has often been [ab]used in a "let CI build/test it so I don't have to" fashion. If it's not part of the tool/action's normal behavior, some people aren't going to bother.

I almost wonder if we should setup a homebrew-core-testing repo or similar that's got some basic macOS (and eventually Linux) testing setup. brew bump-formula-pr should create PRs there and, if CI completes successfully, PRs are autocreated on homebrew-core. This is probably technically difficult (maybe impossible) but it feels like the workflow we want.

[Rylan12]

This strikes me as easier than having to decide what to rename a failed PR to (or having to remember whatever renaming convention we would decide on).

Agreed

This could be a GitHub Actions job that's run nightly and commits a (JSON?) file to formulae.brew.sh to be displayed there.

Yeah, I don't think this would be very difficult at all. And once that JSON file exists its easy to format it however we'd like if we want a human-readable interface (or we can just leave it if we don't).

I almost wonder if we should setup a homebrew-core-testing repo or similar that's got some basic macOS (and eventually Linux) testing setup. brew bump-formula-pr should create PRs there and, if CI completes successfully, PRs are autocreated on homebrew-core. This is probably technically difficult (maybe impossible) but it feels like the workflow we want.

This sounds like a good idea to me. I don't think it should be overly difficult. I think the GitHub actions workflows are flexible enough to do this (on first though). We can have one workflow that creates the PRs. Then, the PR will automatically run the build job to see if it passes. We can then, depending on the outcome of that (those?) jobs, either create a new PR in homebrew/core or close the broken PR and delete the branch.

One question that comes to mind: if we're testing whether they build before opening the PR, can we just generate the bottle then and not duplicate the work in homebrew/core? I know there are other talks about improving the CI jobs that I haven't been very involved in. I wonder, though, if we can detect whether bottles for a given revision exist already and, if so, only run the tests on the formula dependents.

[MikeMcQuaid]

One question that comes to mind: if we're testing whether they build before opening the PR, can we just generate the bottle then and not duplicate the work in homebrew/core? I know there are other talks about improving the CI jobs that I haven't been very involved in. I wonder, though, if we can detect whether bottles for a given revision exist already and, if so, only run the tests on the formula dependents.

Yeh, the more I think about this the more I wonder if this should just be in homebrew-core but we have some automation that detects and closes out brew bump-formula-pr PRs aggressively if they have red CI and no activity e.g. 24-48h after a red status. That would avoid the need for separate tooling for separate repositories.

[samford]

Yeh, the more I think about this the more I wonder if this should just be in homebrew-core but we have some automation that detects and closes out brew bump-formula-pr PRs aggressively if they have red CI and no activity e.g. 24-48h after a red status. That would avoid the need for separate tooling for separate repositories.

Though the separate testing repository idea is interesting, I think keeping it in homebrew-core and taking steps to keep things manageable may be the more reasonable approach in relation to brew bump-formula-pr.

One potentially unexpected side effect of the separate testing repository setup is that PRs in that repo would be associated with the contributor who ran brew bump-formula-pr but, correct me if I'm wrong, we would seemingly have to create the PR in homebrew-core using an account under our control (e.g., BrewTestBot). The commit would still be associated with the contributor but the PR in homebrew-core wouldn't, so we wouldn't be able to see who really opened it at a glance.

That isn't a dealbreaker but I think it would decrease maintainer quality of life a little and the two repository setup may also be confusing to contributors (e.g., "Why is my PR not showing up in homebrew-core? Do I need to run brew bump-formula-pr again?").