Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(core): filter branch in preparation for nx import #27652

Merged
merged 12 commits into from
Aug 29, 2024

Conversation

jaysoo
Copy link
Member

@jaysoo jaysoo commented Aug 27, 2024

This PR fixes an issue with nx import where multiple directories cannot be imported from the same repo. We now use git filter-repo (if installed) or git filter-branch (fallback) to prepare the repo for import so anything not in the subdirectory will be ignored (i.e. cleaner history).

Note: filter-repo is much faster but requires the user to install it first via their package manager (e.g. brew install git-filter-repo or apt install git-filter-repo). We fallback to git filter-branch since it comes with git, but it is slower, so the process may take 10+ minutes on really large monorepos (e.g. next.js).

Also:

  • Use await before returning a promise to Node can maintain correct stacktrace
  • Remove logic for if (relativeSourceDir === '') since using filter-branch moves all the files to the root (.)
  • Default destination project location to be the same as source (e.g. importing packages/a will go to packages/a unless user types in something else)
  • Add --depth option if users don't want to clone with full history (default is full history)
  • Fix issues with special characters causing git ls-files + git mv to since mv doesn't work with escaped names

Current Behavior

Expected Behavior

Related Issue(s)

Fixes #

@jaysoo jaysoo requested a review from a team as a code owner August 27, 2024 15:24
@jaysoo jaysoo requested a review from xiongemi August 27, 2024 15:24
Copy link

vercel bot commented Aug 27, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Updated (UTC)
nx-dev ⬜️ Ignored (Inspect) Visit Preview Aug 29, 2024 7:44pm

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch from 97e1750 to 0df1f8f Compare August 27, 2024 15:25
Copy link

nx-cloud bot commented Aug 27, 2024

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch 7 times, most recently from a405076 to dd30ea1 Compare August 27, 2024 19:18
Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-a405076
Release details 📑
Published version 0.0.0-pr-27652-a405076
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit a405076
Workflow run 10584370515

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@jaysoo jaysoo requested a review from a team as a code owner August 27, 2024 20:39
@jaysoo jaysoo requested a review from isaacplmann August 27, 2024 20:39
Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-0a94c12
Release details 📑
Published version 0.0.0-pr-27652-0a94c12
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit 0a94c12
Workflow run 10585273535

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-a15068b
Release details 📑
Published version 0.0.0-pr-27652-a15068b
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit a15068b
Workflow run 10598165965

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch from a15068b to 3af8043 Compare August 28, 2024 15:39
Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-3af8043
Release details 📑
Published version 0.0.0-pr-27652-3af8043
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit 3af8043
Workflow run 10599985131

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch 2 times, most recently from 5afacbf to 5a954f2 Compare August 28, 2024 19:22
Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-5a954f2
Release details 📑
Published version 0.0.0-pr-27652-5a954f2
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit 5a954f2
Workflow run 10603094689

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch 3 times, most recently from 161cc40 to 8b3f1eb Compare August 29, 2024 01:54
Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-161cc40
Release details 📑
Published version 0.0.0-pr-27652-161cc40
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit 161cc40
Workflow run 10607305410

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch from 8b3f1eb to cc7682b Compare August 29, 2024 13:46
@jaysoo jaysoo force-pushed the fix/import-filter-branch branch from fa7e140 to 6c0bafe Compare August 29, 2024 16:40
packages/nx/src/command-line/import/import.ts Outdated Show resolved Hide resolved
packages/nx/src/utils/git-utils.ts Outdated Show resolved Hide resolved
@jaysoo jaysoo force-pushed the fix/import-filter-branch branch from 1746961 to ef19c0f Compare August 29, 2024 18:06
Copy link

🐳 We have a release for that!

This PR has a release associated with it. You can try it out using this command:

npx [email protected] my-workspace

Or just copy this version and use it in your own command:

0.0.0-pr-27652-1746961
Release details 📑
Published version 0.0.0-pr-27652-1746961
Triggered by @jaysoo
Branch fix/import-filter-branch
Commit 1746961
Workflow run 10620171971

To request a new release for this pull request, mention someone from the Nx team or the @nrwl/nx-pipelines-reviewers.

@jaysoo jaysoo force-pushed the fix/import-filter-branch branch 3 times, most recently from f8ea88a to 8b04549 Compare August 29, 2024 19:26
@jaysoo jaysoo force-pushed the fix/import-filter-branch branch from 8b04549 to 5d5aa16 Compare August 29, 2024 19:41
@FrozenPandaz FrozenPandaz merged commit ab408ab into master Aug 29, 2024
4 of 6 checks passed
@FrozenPandaz FrozenPandaz deleted the fix/import-filter-branch branch August 29, 2024 20:38
FrozenPandaz pushed a commit that referenced this pull request Sep 3, 2024
This PR fixes an issue with `nx import` where multiple directories
cannot be imported from the same repo. We now use `git filter-repo` (if
installed) or `git filter-branch` (fallback) to prepare the repo for
import so anything not in the subdirectory will be ignored (i.e. cleaner
history).

Note: `filter-repo` is much faster but requires the user to install it
first via their package manager (e.g. `brew install git-filter-repo` or
`apt install git-filter-repo`). We fallback to `git filter-branch` since
it comes with git, but it is slower, so the process may take 10+ minutes
on really large monorepos (e.g. next.js).

Also:
- Use `await` before returning a promise to Node can maintain correct
stacktrace
- Remove logic for `if (relativeSourceDir === '')` since using
`filter-branch` moves all the files to the root (`.`)
- Default destination project location to be the same as source (e.g.
importing `packages/a` will go to `packages/a` unless user types in
something else)
- Add `--depth` option if users don't want to clone with full history
(default is full history)
- Fix issues with special characters causing `git ls-files` + `git mv`
to since `mv` doesn't work with escaped names

<!-- If this is a particularly complex change or feature addition, you
can request a dedicated Nx release for this pull request branch. Mention
someone from the Nx team or the `@nrwl/nx-pipelines-reviewers` and they
will confirm if the PR warrants its own release for testing purposes,
and generate it for you if appropriate. -->

## Current Behavior
<!-- This is the behavior we have today -->

## Expected Behavior
<!-- This is the behavior we should expect with the changes in this PR
-->

## Related Issue(s)
<!-- Please link the issue being fixed so it gets closed when this is
merged. -->

Fixes #

(cherry picked from commit ab408ab)
Copy link

github-actions bot commented Sep 4, 2024

This pull request has already been merged/closed. If you experience issues related to these changes, please open a new issue referencing this pull request.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 4, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants