BUG: jj checkout and HEAD@git out of sync

[shavitmichael]

jj can enter a state where its currently checked out commit get's out of sync with the git HEAD in a colocated repository. This looks similar to #1495 but doesn't seem to involve unborn branches (AFAICT).

My jj log looks like:

❯ jj log
@  zmvmkzxk [email protected] 2023-08-09 00:30:18.000 +08:00 393d6c13
│  (empty) (no description set)
◉  xxttkkus [email protected] 2023-08-04 02:11:47.000 +08:00 HEAD@git c09cbd8e
│  iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
◉  xqmpkklm [email protected] 2023-08-04 02:11:46.000 +08:00 2279e212
│  iommu/arm-smmu-v3: Free pasid domains on iommu release

and git status

❯ git checkout HEAD~1
Previous HEAD position was c09cbd8ed86ef iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
HEAD is now at 2279e21297f5c iommu/arm-smmu-v3: Free pasid domains on iommu release

This looks ok at a glance, but updating the currently checked out commit in either jj or git is broken

Checking out a commit in git should be reflected in jj with an updated HEAD@git label, and a new checked-out commit @. Neither of those get updated however.

❯ git checkout HEAD~1
Previous HEAD position was c09cbd8ed86ef iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
HEAD is now at 2279e21297f5c iommu/arm-smmu-v3: Free pasid domains on iommu release
❯ jj log
@  npytpmkq [email protected] 2023-08-09 00:40:15.000 +08:00 10f210ed
│  (no description set)
◉  xxttkkus [email protected] 2023-08-04 02:11:47.000 +08:00 HEAD@git c09cbd8e
│  iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
◉  xqmpkklm [email protected] 2023-08-04 02:11:46.000 +08:00 2279e212
│  iommu/arm-smmu-v3: Free pasid domains on iommu release

Similarly, updating the checked out commit in jj should be reflected on the git side; but git is instead still checked out at the previous commit and thus sees changes in jj's checked out commit as diffs.

// Undo previous git checkout for a cleaner experiment
❯ git checkout c09cbd8ed86ef
Previous HEAD position was 2279e21297f5c iommu/arm-smmu-v3: Free pasid domains on iommu release
HEAD is now at c09cbd8ed86ef iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
❯  jj co xqmpkklm
❯ jj log
@  wvnrupnq [email protected] 2023-08-09 00:42:05.000 +08:00 18804d9c
│  (empty) (no description set)
│ ◉  xxttkkus [email protected] 2023-08-04 02:11:47.000 +08:00 HEAD@git c09cbd8e
├─╯  iommu/arm-smmu-v3: Cleanup arm_smmu_domain_finalise
◉  xqmpkklm [email protected] 2023-08-04 02:11:46.000 +08:00 2279e212
│  iommu/arm-smmu-v3: Free pasid domains on iommu release
❯ git status
HEAD detached at c09cbd8ed86ef
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c

I haven't yet figured out how I got into this state, let alone reproduction steps. Let me know if you have any suggestions on how I could investigate further.

[martinvonz]

Thanks for the report.

Maybe you can get some hints from running jj op log. You should see some operations saying "import git refs". You can also try jj debug operation <operation id> for various operations there and see where the git_head and wc_commit_ids point. Let us know what you find.

[ilyagr]

Also, what version of jj are you using? Is it 0.8? Newer unreleased version? jj version should tell you the exact commit.

[shavitmichael]

Also, what version of jj are you using? Is it 0.8? Newer unreleased version? jj version should tell you the exact commit.

Running on jj 0.8.0-b6c65c12aca94c0574b629424931ea7f772babed

Maybe you can get some hints from running jj op log. You should see some operations saying "import git refs". You can also try jj debug operation <operation id> for various operations there and see where the git_head and wc_commit_ids point. Let us know what you find.

1. Git->jj direction

Ok, trying out the git checkout HEAD~1 test, and the first possibly unexpected observation is that jj log does not result in an "import git refs" operation:

@  553776c13193 [email protected] 15 seconds ago, lasted 1 second
│  snapshot working copy
│  args: jj log

The same operation on a fresh checkout of the repo (also collocated) has a different behavior

@  3c0a985116c0 [email protected] 7 seconds ago, lasted 2 seconds                                          │ git branch <new-branch-name> 149ba7d0c399
│  import git refs                                                                                                           │
│  args: jj log  

Perhaps expected then, the git_head isn't updated, and the wc points to an (empty) wc child of the stale git_head

    git_head: RefTarget {
        conflict: Conflict {
            removes: [],
            adds: [
                Some(
                    CommitId(
                        "c09cbd8ed86ef376fba5ee842a9d1925eaff6644",
                    ),
                ),
            ],
        },
    },
    wc_commit_ids: {
        WorkspaceId(
            "default",
        ): CommitId(
            "bf24c5c944fe0b3a20f60c12009fcfe1d2832718",
        ),
    },

Explicitly calling jj git import correctly updates the git_head in jj to reflect the one in git, but wc_commit_ids isn't updated.

2. jj->Git

Running jj co @-- doesn't update the git_head:

❯ jj op log

@  07ae677156ae [email protected] 8 seconds ago, lasted 1 second
│  check out commit 2279e21297f5c8540810288a2d3023c2b7ff5b94
│  args: jj co @--

❯ jj debug operation @

    git_head: RefTarget {
        conflict: Conflict {
            removes: [],
            adds: [
                Some(
                    CommitId(
                        "c09cbd8ed86ef376fba5ee842a9d1925eaff6644",
                    ),
                ),
            ],
        },
    },

In this direction, jj git export has no effect.

❯ jj git export
Nothing changed.

Meanwhile, the same operation on the fresh checkout does update the git_head.

@  c639e2dba547 [email protected] 21 seconds ago, lasted 1 second
│  check out commit d92b86f672a42d9d74a24a63a1e59793c4116830
│  args: jj co @--

    git_head: RefTarget {
        conflict: Conflict {
            removes: [],
            adds: [
                Some(
                    CommitId(
                        "d92b86f672a42d9d74a24a63a1e59793c4116830",
                    ),
                ),
            ],
        },
    },

[shavitmichael]

Going further back the operation log, the repo was initially OK as changes were only being made on the Git side and jj was correctly tracking changes to the git_head on each call to jj log. The first jj checkout operation ever performed on the repo did not update the git_head, and I suspect things just got worse from there. The first such operation is shown below:

**jj op log**
◉  893f3050d14f [email protected] 6 days ago, lasted 775 milliseconds
│  check out commit 46b367e9640a74cb57e9555eaa78328162adaa31
│  args: jj co patman
◉  d7b3ef476cfb [email protected] 6 days ago, lasted 1 second
│  snapshot working copy
│  args: jj log

## Everything ok at and before d7b3ef476cfb

**jj log --at-operation d7b3ef476cfb**
@  rousutqr (no name configured) <(no email configured)> 2023-08-03 10:43:57.000 +08:00 19ba21da
│  (empty) (no description set)
◉  ktnrkutt Michael Shavit <[email protected]> 2023-08-03 02:10:54.000 +08:00 patman_upload HEAD@git b39e2a81
│  smmu test-engine fix

**  jj debug operation d7b3ef476cfb**
    git_head: RefTarget {
        conflict: Conflict {
            removes: [],
            adds: [
                Some(
                    CommitId(
                        "b39e2a81732d3c48071aef990052de9c66af136f",
                    ),
                ),
            ],
        },
    },
    wc_commit_ids: {
        WorkspaceId(
            "default",
        ): CommitId(
            "19ba21da1a131186d997bd06c216d368ef3f37bd",
        ),
    },

## Now things go wrong at 893f3050d14f

** jj log --at-operation 893f3050d14f**
@  omzqkxwu (no name configured) <(no email configured)> 2023-08-03 10:59:23.000 +08:00 6c9deb0f
│  (empty) (no description set)
◉  rnuuouvt Michael Shavit <[email protected]> 2023-08-03 02:07:23.000 +08:00 patman 46b367e9
│  smmu test-engine fix

## (Note: "smmu test-engine fix" 46b367e9 has a different sha/change-id than b39e2a81)*

**  jj debug operation 893f3050d14f**
    git_head: RefTarget {
        conflict: Conflict {
            removes: [],
            adds: [
                Some(
                    CommitId(
                        "b39e2a81732d3c48071aef990052de9c66af136f",
                    ),
                ),
            ],
        },
    },
    wc_commit_ids: {
        WorkspaceId(
            "default",
        ): CommitId(
            "6c9deb0f6823b9c72f011052363795bf061275b7",
        ),
    },
}

I did early on also create a non-collocated jj workspace backed by the same repo; could that have confused the collocated one?

[yuja]

I did early on also create a non-collocated jj workspace backed by the same repo; could that have confused the collocated one?

The check for "colocated" repo is implemented as follows. I'm not pretty sure, but if Git's worktree is involved, weird thing might occur.
https://github.com/martinvonz/jj/blob/b6c65c12aca94c0574b629424931ea7f772babed/src/cli_util.rs#L695-L700
(note: workspace_root() is canonicalized)

[shavitmichael]

Thanks for the pointer. Added some hacky prints

        if let Some(git_workdir) = maybe_git_backend
            .and_then(|git_backend| {tracing::info!("maybe_git_backend"); git_backend.git_repo().workdir().map(ToOwned::to_owned)})
            .and_then(|workdir| {tracing::info!("mapped");workdir.canonicalize().ok()})
        {
            working_copy_shared_with_git = git_workdir == workspace.workspace_root().as_path();
            tracing::info!(git_workdir=git_workdir.to_str(), workspace_root=workspace.workspace_root().as_path().to_str());  
        }
        tracing::info!(wc_shared_with_git=working_copy_shared_with_git);

Which results in

❯ jj co c09cbd8ed86e --verbose
2023-08-09T15:43:36.268691Z  INFO jj_cli::cli_util: verbose logging enabled
2023-08-09T15:43:36.353494Z  INFO run_command:cmd_checkout:workspace_helper_internal{snapshot=true}:for_loaded_repo:new: jj_cli::cli_util: maybe_git_backend
2023-08-09T15:43:36.353514Z  INFO run_command:cmd_checkout:workspace_helper_internal{snapshot=true}:for_loaded_repo:new: jj_cli::cli_util: mapped
2023-08-09T15:43:36.353545Z  INFO run_command:cmd_checkout:workspace_helper_internal{snapshot=true}:for_loaded_repo:new: jj_cli::cli_util: git_workdir="<repo_root>/.repo/projects/linux/" workspace_root="<repo_root>/linux/common"
2023-08-09T15:43:36.353564Z  INFO run_command:cmd_checkout:workspace_helper_internal{snapshot=true}:for_loaded_repo:new: jj_cli::cli_util: working_copy_shared_with_git=false

Notably, this git repo is managed by the repo tool, with jj initialized. I think we can conclude that this issue was caused by initializing jj in a git repo managed by the repo tool.

I haven't looked too deep at what repo is doing under the hood yet, but <repo_root>/linux/common appears to be neither a symlink nor a git worktree:

From <repo_root>/linux/common (where jj init --git-repo=. and all of my development was made from)

❯ git worktree list -v
<repo_root>/.repo/projects/linux/common.git  c09cbd8ed86ef (detached HEAD)

[shavitmichael]

I haven't looked too deep at what repo is doing under the hood yet, but <repo_root>/linux/common appears to be neither a symlink nor a git worktree:

Oh, the .git directory is a symlink to the common.git folder found in .repo .

[martinvonz]

Interesting. Maybe we should add a config controlling whether we consider the workspace colocated. Then you would be able to override it in your case. We could set the config when we initialize the workspace. I don't know if there's a better solution.

[shavitmichael]

(Can confirm that forcing working_copy_shared_with_git = true resolved all the issues from the original comment.)

[shavitmichael]

That was quick; tried out your fix and it worked :). Thanks Yuya!