Panic while executing jj log

[mhartkorn]

Description

I was using jj to track a new project and suddenly it started to output the following panic message:

$ RUST_BACKTRACE=full jj
thread 'main' panicked at cli\src\cli_util.rs:1753:14:
called `Result::unwrap()` on an `Err` value: NotFound
stack backtrace:
   0:     0x7ff7f138090a - git_odb_object_size
   1:     0x7ff7f13a608b - git_tree_entry_filemode_raw
   2:     0x7ff7f137a971 - git_odb_object_size
   3:     0x7ff7f138068a - git_odb_object_size
   4:     0x7ff7f13835ea - git_odb_object_size
   5:     0x7ff7f1383258 - git_odb_object_size
   6:     0x7ff7f1383c9e - git_odb_object_size
   7:     0x7ff7f1383b8d - git_odb_object_size
   8:     0x7ff7f13815c9 - git_odb_object_size
   9:     0x7ff7f1383890 - git_odb_object_size
  10:     0x7ff7f1543f75 - git_midx_writer_new
  11:     0x7ff7f15443d4 - git_midx_writer_new
  12:     0x7ff7f0d7be7f - git_filter_source_repo
  13:     0x7ff7f0d77e4b - git_filter_source_repo
  14:     0x7ff7f0d7591b - git_filter_source_repo
  15:     0x7ff7f0d6a9a0 - git_filter_source_repo
  16:     0x7ff7f0e2bc43 - git_filter_source_repo
  17:     0x7ff7f0c1cb83 - <unknown>
  18:     0x7ff7f0c8b74b - <unknown>
  19:     0x7ff7f0d84f05 - git_filter_source_repo
  20:     0x7ff7f0bd106e - <unknown>
  21:     0x7ff7f0bd1006 - <unknown>
  22:     0x7ff7f1371548 - git_odb_object_size
  23:     0x7ff7f0bd10ac - <unknown>
  24:     0x7ff7f1514ab8 - git_midx_writer_new
  25:     0x7ffbcdc17344 - BaseThreadInitThunk
  26:     0x7ffbcde626b1 - RtlUserThreadStart

I didn't create new branches or play with advanced features.

One thing I did out of the ordinary was using a colocated git repo to be able to see changes in IntelliJ and set the default for jj to show jj log.

I accidentally tried commiting a >1MiB file but was warned about this by jj and then added the file to .gitignore.

Steps to Reproduce the Problem

I honestly don't know.

Expected Behavior

Show the log

Actual Behavior

Error, see above.

Specifications

• Platform: Windows 10 22H2 19045.3570 x86-64
• Version: jj 0.11.0-f00f7527ddce21886fe93e11324c5227f15eeaca

[yuja]

I have no idea why the backtrace contains something named git, but the panicking location is here:
https://github.com/martinvonz/jj/blob/f00f7527ddce21886fe93e11324c5227f15eeaca/cli/src/cli_util.rs#L1750-L1753

It implies that the file named .jj/repo/op_store/operations/{op_id} is somehow lost. (it might be filesystem or antivirus issue.)
The op_id can be obtained from jj debug workingcopy. It will be printed at top.

I don't know if there's a simple recovery method. Maybe you can create a fresh new workspace by jj workspace add <new-workspace-path> --ignore-working-copy and discard the current panicking workspace (or copy back .jj/working_copy files.) If you don't care jj's operation history, recreating the .jj repository would be the easiest option.

[martinvonz]

I wonder if this is related to the flakiness we see in test_concurrent_operations.rs on Windows and the flakiness reported in #2103. Maybe we need to add some system call to flush writes.

[mhartkorn]

It implies that the file named .jj/repo/op_store/operations/{op_id} is somehow lost. (it might be filesystem or antivirus issue.) The op_id can be obtained from jj debug workingcopy. It will be printed at top.

jj debug workingcopy is printing the exact same error.

I don't know if there's a simple recovery method. Maybe you can create a fresh new workspace by jj workspace add <new-workspace-path> --ignore-working-copy and discard the current panicking workspace (or copy back .jj/working_copy files.) If you don't care jj's operation history, recreating the .jj repository would be the easiest option.

Recreating the repo helped and it works now, thank you. I saved the old .jj directory if anybody wants to debug this further but for me this is solved.

[yuja]

jj debug workingcopy is printing the exact same error.

Right. jj debug workingcopy --ignore-working-copy will probably work.

[aymanrady]

I'm not sure if this is related to the same issue reported here or if I should open an new issue, but I get a panic when running RUST_BACKTRACE=full jj log -r 'diff_contains("customer_id")'

thread 'main' panicked at lib/src/default_index/revset_engine.rs:1131:22:
called `Result::unwrap()` on an `Err` value: ObjectNotFound { object_type: "file", hash: "daa3343fa71d87bf8646b6192768e9883a537a20", source: NotFound { oid: Sha1(daa3343fa71d87bf8646b6192768e9883a537a20) } }
stack backtrace:
   0:     0x7fd3bbaed015 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h706c9783aace6373
   1:     0x7fd3bbb3b24b - core::fmt::write::h7221bf2763310748
   2:     0x7fd3bbae81af - std::io::Write::write_fmt::h5f543a3ea7d45c3d
   3:     0x7fd3bbaecdee - std::sys_common::backtrace::print::hf2079bcf21b5d607
   4:     0x7fd3bbaee8c9 - std::panicking::default_hook::{{closure}}::hd6a41ed4ba4d9cdd
   5:     0x7fd3bbaee66a - std::panicking::default_hook::h2becca4d115c088a
   6:     0x7fd3bbaeed63 - std::panicking::rust_panic_with_hook::he62aa99a29ab1fe6
   7:     0x7fd3bbaeec44 - std::panicking::begin_panic_handler::{{closure}}::h4295830cbed5c147
   8:     0x7fd3bbaed4d9 - std::sys_common::backtrace::__rust_end_short_backtrace::h29724adceafc0e2c
   9:     0x7fd3bbaee977 - rust_begin_unwind
  10:     0x7fd3babf4023 - core::panicking::panic_fmt::h5822797b7c57c9a6
  11:     0x7fd3babf4516 - core::result::unwrap_failed::h79cc1ac0d8b6c33d
  12:     0x7fd3bb1be814 - jj_lib::default_index::revset_engine::build_predicate_fn::{{closure}}::h2d0249d3d3fec6e4
  13:     0x7fd3bb1af85b - <jj_lib::default_index::rev_walk::FilterRevWalk<W,P> as jj_lib::default_index::rev_walk::RevWalk<I>>::next::h7ac70369cf98cd5b
  14:     0x7fd3bb1bf9fc - jj_lib::default_index::revset_graph_iterator::RevsetGraphWalk::new_edges_from_internal_commit::he4a247b5324ae725
  15:     0x7fd3bb1bf626 - jj_lib::default_index::revset_graph_iterator::RevsetGraphWalk::edges_from_internal_commit::h861262572fe22390
  16:     0x7fd3bb1c148f - <jj_lib::default_index::revset_graph_iterator::RevsetGraphWalk as jj_lib::default_index::rev_walk::RevWalk<jj_lib::default_index::composite::CompositeIndex>>::next::h8ec930b456d9bedc
  17:     0x7fd3bb060588 - <core::iter::sources::from_fn::FromFn<F> as core::iter::traits::iterator::Iterator>::next::h8f098cd056ec1b0d
  18:     0x7fd3bacedbb1 - jj_lib::graph::TopoGroupedGraphIterator<N,I>::populate_one::h9d8fffa170559617
  19:     0x7fd3bac0d303 - <jj_lib::graph::TopoGroupedGraphIterator<N,I> as core::iter::traits::iterator::Iterator>::next::h609c69ad1e606c19
  20:     0x7fd3baf21ee3 - jj_cli::commands::run_command::hdcb6797722683dfb
  21:     0x7fd3bac528f0 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h27cc50fc4d51675a
  22:     0x7fd3bad10595 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::haa7f287db3aa1604
  23:     0x7fd3bae61a97 - jj_cli::cli_util::CliRunner::run::h19c9a5ecdc7c05b5
  24:     0x7fd3babf4b41 - jj::main::h217ea02a4cfe6e6b
  25:     0x7fd3babf4ad3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h551a444895aa8204
  26:     0x7fd3babf4ae9 - std::rt::lang_start::{{closure}}::hcf29df5888f4c0b9
  27:     0x7fd3bbaddd50 - std::rt::lang_start_internal::ha0cc8e9091dbafdb
  28:     0x7fd3babf4b75 - main

tried using different values for diff_contains() with the same result, despite other revsets working fine, ::@, description("customer"), and default revset.

jj 0.21.0-d002a5ad35e624a731e96e85f490f28febc7797e on linux

[martinvonz]

I suspect the object really doesn't exist. Did you clone the repo using shallow or partial clone? Does git fsck in the repo pass (git --git-dir .jj/repo/store/git fsck if you're not in a colocated repo)?

[aymanrady]

The repo is colocated, this particular instance has been here since the start of the repository ~5 years ago.

git fsck

broken link from    tree 717512d286728cf1250ed05a3a1c39ffb8627a45
              to    blob daa3343fa71d87bf8646b6192768e9883a537a20
broken link from    tree 48ab26afa443f06434ead6356109d04e06240404
              to    blob d00fc8cd2893183d81e56c5c350699bec0cb82fd
broken link from    tree 48ab26afa443f06434ead6356109d04e06240404
              to    blob c27852f1b7cb6895e8bff234ba1f90abae32b97a
broken link from    tree 6bf6b1f3a9a2d66711950c79016ad05cf9fd2cfc
              to    blob fedb90506c39ac67b4a688a37f24a62a99cc74fc
... <some ~2840 dangling blobs & commits>
Checking object directories: 100% (256/256), done.
Checking objects: 100% (72508/72508), done.
Verifying commits in commit graph: 100% (3196/3196), done.

I'm guessing the broken links are the issue, tried running the same jj log operation on a new clone and it works fine.