Run only a subset of tests on PRs that only modify the manual

[DilumAluthge]

If a PR only modifies the manual, then we only need to run a subset of jobs, including:

1. Build a regular (non-assert) build of Julia on x86_64-linux-gnu
2. Build the docs
3. Run the doctests

[giordano]

And PRs editing only the readme shouldn't run anything

[DilumAluthge]

Cross-ref JuliaLang/julia#45744

[staticfloat]

I think the way to do this is to have a "filter" that looks at the diff, and skips launching a job if the diff does not contain changes to a particular group of files. We can also construct this negatively, e.g. filter out elements of a diff, and then run if any diff remains.

The way I'm imagining this is to write a buildkite plugin that does something like the following:

steps:
  - label: launch jobs
    plugins:
      - JuliaCI/diff-filter#v1:
          docs:
            # Only run docs if we have a diff that changes something in `doc/`
            mode: positive
            patterns:
              - "doc/**"
          build:
            # Run build if anything changes, excluding changes to docs, README, etc...
            mode: negative
            patterns:
              - "doc/**"
              - "README.md"
    commands: |
      if [[ $${BUILDKITE_PLUGIN_DIFF_FILTER_TRIGGERED_DOCS} == 1 ]]; then
          buildkite-agent upload docs.yml
      fi
      if [[ $${BUILDKITE_PLUGIN_DIFF_FILTER_TRIGGERED_BUILD} == 1 ]]; then
          buildkite-agent upload build.yml
      fi

[staticfloat]

To filter the build, we can use this code: https://github.com/staticfloat/forerunner-buildkite-plugin/blob/856c0f5d49a5d2d84270cb14b41cbb4fa5d3dc8b/hooks/command#L8-L9 and https://github.com/staticfloat/forerunner-buildkite-plugin/blob/856c0f5d49a5d2d84270cb14b41cbb4fa5d3dc8b/hooks/command#L59-L100

[giordano]

In Yggdrasil we collect the files modified in a PR/commit: https://github.com/JuliaPackaging/Yggdrasil/blob/fb8152880d5bb011c4ab2a23b84fff2327c1ee17/azure-pipelines.yml#L81-L86. Valentin will have done something similar with buildkite in JuliaPackaging/Yggdrasil#4771

[fatteneder]

I put together a plugin based on @staticfloat idea: https://github.com/fatteneder/diff-filter-buildkite-plugin

It seems to work fine in my tests: https://buildkite.com/testingplugin/bash-example/builds/63#0192d94f-a732-4f76-bb12-ad2262b75cc9

How would I go about testing it with the Julia repo?
Just open a PR here?

[giordano]

It seems to work fine in my tests: buildkite.com/testingplugin/bash-example/builds/63#0192d94f-a732-4f76-bb12-ad2262b75cc9

I get "page not found"

How would I go about testing it with the Julia repo?
Just open a PR here?

Think so.

[fatteneder]

I get "page not found"

My bad, pipeline was private, should work now.