Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for partial checkout filters #1396

Merged
merged 9 commits into from
Sep 22, 2023

Conversation

finleygn
Copy link
Contributor

@finleygn finleygn commented Jun 30, 2023

A change to add support for sparse checkouts has been added recently (#1369). This extends upon that to allow support for other filters, and the ability to filter without a sparse checkout.

- uses: actions/checkout@v3
  with:
    filter: tree:0

Inspired by this blog post -> https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone

@finleygn finleygn changed the title added filter option & tests Add "filter" option Jun 30, 2023
@finleygn finleygn changed the title Add "filter" option Add "filter" fetch option Jun 30, 2023
@finleygn finleygn changed the title Add "filter" fetch option Add support for partial checkout filters Jun 30, 2023
@finleygn finleygn marked this pull request as ready for review June 30, 2023 16:32
@finleygn finleygn requested a review from a team as a code owner June 30, 2023 16:32
@gabrielyotoo
Copy link

bump +1 this would be nice to use with Yarn

@Rycieos
Copy link

Rycieos commented Jul 19, 2023

This is an often requested feature:
#172, #663, #1152

I would personally love to see this added.

@petergaultney
Copy link

i've used this fork/branch successfully with both tree:0 and blob:none. Thank you!

@flying-sheep
Copy link

I think filter: 'tree:0' would be a much better default than fetch-depth: 1.

It would be almost exactly as small, but without breaking the Git history.

CLONE_FILTER=$(git config --local --get remote.origin.partialclonefilter)

if [ "$CLONE_FILTER" != "blob:none" ]; then
echo "Expected ./fetch-filter/.git/config to have 'remote.origin.partialclonefilter' set to 'blob:none'"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you miss exit 1 here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, good spot 😳

@jakobmerrild
Copy link

This is exactly what we need in our repo. Speeds up checkout by a factor of 4 without breaking git history so our existing logic that uses git diff and git blame will still work ❤️

@cory-miller
Copy link
Contributor

Thanks for your contribution.

Do you mind adding an update to https://github.com/actions/checkout/blob/72f2cec99f417b1a1c5e2e88945068983b7965f9/action.yml#L56C19-L60 with the new setting with a note that it will override the sparseCheckout option if also set?

The rest LGTM.

@finleygn
Copy link
Contributor Author

Will take a look as to whats causing the test to fail soon...

@finleygn
Copy link
Contributor Author

@cory-miller the test actions should pass now 🙏

@cory-miller cory-miller merged commit c533a0a into actions:main Sep 22, 2023
11 checks passed
@cory-miller
Copy link
Contributor

Thanks for working on this!

I will build a release soon.

@cory-miller
Copy link
Contributor

actions/checkout@v4 or actions/[email protected] are updated with this change.

@Rycieos
Copy link

Rycieos commented Sep 22, 2023

@finleygn thanks for working on this!

@cory-miller the issues #172, #663, and #1152 should likely be closed.

@benoit-pierre
Copy link

This does not seem to apply to submodules.

@Kurt-von-Laven
Copy link

I think filter: 'tree:0' would be a much better default than fetch-depth: 1.

It would be almost exactly as small, but without breaking the Git history.

This is a cool feature, but please don't make the default checkout larger for everyone since most users don't need the Git history in their CI pipelines. We do often need the history, so we tried switching from fetch-depth: 0 to filter: tree:0, and our CI pipelines became twice as slow overall. Then we tried filter: blob:none, and that still slowed down our CI pipelines by 50%. I suspect this is happening because some tool (e.g., Gitleaks) is causing Git to fetch the missing data piecemeal rather than more efficiently all at once by attempting to access trees or blobs that aren't available locally. I don't mean this as a knock on partial clones; everyone should always measure the impact of performance changes in their own specific use case. I would guess that treeless clones are a net performance benefit if and only if no trees or blobs from previous commits are needed, because GitHub has already put so much effort into optimizing checkout times in GitHub Actions. Similarly, I would guess that blobless clones are a net performance benefit if and only if no blobs from previous commits are needed.

@jakebailey
Copy link

jakebailey commented Sep 30, 2023

I think there's something missing from the docs here; you still need to explicitly set fetch-depth: 0 in order to get all of history, making a full workflow look like:

- use: actions/checkout@v4
  with:
    fetch-depth: 0
    filter: blob:none

Without setting it, you still only get the most recent commit.

For the TypeScript repo:

  • fetch-depth: null, filter: null: 12s, but only one commit.
  • fetch-depth: null, filter: blob:none: 30s, but still only one commit!
  • fetch-depth: 0, filter: null, 4m6s: full history
  • fetch-depth: 0, filter: blob:none, 1m37s: full history

For example, https://github.com/microsoft/TypeScript/actions/runs/6359689867/job/17272959471?pr=55923 (though this link will eventually break)

@Kurt-von-Laven
Copy link

@jakebailey, did you mean filter: blob:none rather than filter: filter:none for the last bullet?

@jakebailey
Copy link

Yes, that's just a typo on my part.

gitfool added a commit to gitfool/cake-docker that referenced this pull request Oct 21, 2023
gitfool added a commit to gitfool/Cake.Dungeon that referenced this pull request Oct 21, 2023
gitfool added a commit to gitfool/GitHubActions.Dungeon that referenced this pull request Oct 21, 2023
jsuchenia pushed a commit to jsuchenia/adventofcode that referenced this pull request Dec 2, 2023
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/checkout](https://github.com/actions/checkout) | action | major | `v3` -> `v4` |

---

### Release Notes

<details>
<summary>actions/checkout (actions/checkout)</summary>

### [`v4`](https://github.com/actions/checkout/blob/HEAD/CHANGELOG.md#v410)

[Compare Source](actions/checkout@v3...v4)

-   [Add support for partial checkout filters](actions/checkout#1396)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4zMS43IiwidXBkYXRlZEluVmVyIjoiMzcuMzEuNyIsInRhcmdldEJyYW5jaCI6Im1hc3RlciJ9-->

Reviewed-on: https://git.apud.pl/jacek/adventofcode/pulls/38
Co-authored-by: Renovate <[email protected]>
Co-committed-by: Renovate <[email protected]>
gitfool added a commit to gitfool/Cake.Dungeon that referenced this pull request Jan 7, 2024
gitfool added a commit to gitfool/BoardGameGeek.Dungeon that referenced this pull request Jan 7, 2024
gitfool added a commit to gitfool/Pulumi.Dungeon that referenced this pull request Jan 13, 2024
gitfool added a commit to gitfool/Pulumi.Dungeon that referenced this pull request Jan 13, 2024
JoelDewey pushed a commit to JoelDewey/rhel-wireguard that referenced this pull request Jan 23, 2024
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/checkout](https://github.com/actions/checkout) | action | major | `v3` -> `v4` |

---

### Release Notes

<details>
<summary>actions/checkout (actions/checkout)</summary>

### [`v4`](https://github.com/actions/checkout/blob/HEAD/CHANGELOG.md#v410)

[Compare Source](actions/checkout@v3...v4)

-   [Add support for partial checkout filters](actions/checkout#1396)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xNDEuMCIsInVwZGF0ZWRJblZlciI6IjM3LjE0MS4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

Reviewed-on: https://gitea.int.magicalinternetbox.net/joel/rhel-wireguard/pulls/10
Co-authored-by: Renovate Bot <[email protected]>
Co-committed-by: Renovate Bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.