Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

isfile_casesensitive fixes on Windows #55220

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Jul 23, 2024

  • Expand longpath error info, otherwise you just get
  SystemError: longpath: The system cannot find the path specified. 
  Stacktrace:
    [1] windowserror(p::Symbol, code::UInt32; extrainfo::Nothing)
      @ Base .\error.jl:191
    [2] windowserror
      @ .\error.jl:190 [inlined]
    [3] longpath(path::String)
      @ Base.Filesystem .\path.jl:479
  • Use abspath in longpath because GetLongPathNameW doesn't seem to handle relative paths

Fixes #54799

cc. @kimikage

@IanButterworth IanButterworth changed the title add path to longpath error for context isfile_casesensitive fixes on Windows Jul 23, 2024
@IanButterworth IanButterworth added the needs tests Unit tests are required for this change label Jul 23, 2024
@IanButterworth
Copy link
Member Author

More info on the Pkg test case we see this on

C:\Users\runneradmin\AppData\Local\Temp\jl_bMlLt2\environments\v1.12\..\..\..\jl_vAJCF9\B
cyclic dependency graph: Error During Test at D:\a\Pkg.jl\Pkg.jl\test\new.jl:2935
  Got exception outside of a @test
  SystemError: longpath: The system cannot find the path specified. 
  Stacktrace:
    [1] windowserror(p::Symbol, code::UInt32; extrainfo::Nothing)
      @ Base .\error.jl:200
    [2] windowserror
      @ .\error.jl:199 [inlined]
    [3] longpath(path::String)
      @ Base.Filesystem .\path.jl:479
    [4] isfile_casesensitive(path::String)
      @ Base .\loading.jl:15

@IanButterworth
Copy link
Member Author

The issue seems to be that longpath didn't like relative paths

@IanButterworth IanButterworth added backport 1.10 Change should be backported to the 1.10 release backport 1.11 Change should be backported to release-1.11 and removed needs tests Unit tests are required for this change labels Jul 23, 2024
@stevengj
Copy link
Member

Might be worth perusing the discussion of the original implementation of isfile_casesensitive #13542 and a subsequent patch on Windows #19291.

@stevengj
Copy link
Member

LGTM. The Win32 docs on GetLongPathNameW aren't clear on this point to me, but other discussions online seem to agree that its argument should be an absolute path.

@IanButterworth

This comment was marked as resolved.

@IanButterworth
Copy link
Member Author

Maybe other platforms need abspath too? This is on MacOS

julia> touch("newfile")
"newfile"

julia> Base.isfile_casesensitive("newfile")
true

julia> Base.isfile_casesensitive(joinpath("dummy", "..", "newfile"))
false

@stevengj
Copy link
Member

Looks like on MacOS the culprit may be isaccessiblefile, which is called by isfile_casesensitive before doing anything else:

julia> touch("newfile")
"newfile"

julia> Base.isaccessiblefile("newfile")
true

julia> Base.isaccessiblefile("dummy/../newfile")
false

@IanButterworth
Copy link
Member Author

IanButterworth commented Jul 23, 2024

If my relative path is valid, in that it steps in and out of a real dir, then

julia> Base.isfile_casesensitive("src/../newfile")
true

julia> Base.isfile_casesensitive("dummy/../newfile")
false

@stevengj
Copy link
Member

stevengj commented Jul 23, 2024

Looks like both isaccessiblefile and ccall(:getattrlist, ...) (both are used inside isfile_casesensitive) don't like the relative path with the dummy component.

Given that isfile also fails for a dummy component:

julia> isfile("dummy/../newfile")
false

it's not obvious to me that this behavior is wrong?

@stevengj
Copy link
Member

stevengj commented Jul 23, 2024

On Windows, does the old isfile_casesensitive succeed for relative paths involving existing directories, or is abspath needed there too?

i.e. is this PR only for dummy path components?

test/loading.jl Outdated Show resolved Hide resolved
@IanButterworth
Copy link
Member Author

Yeah, added tests here and testing master in #55228

@KristofferC KristofferC mentioned this pull request Jul 24, 2024
46 tasks
KristofferC added a commit that referenced this pull request Aug 13, 2024
Backported PRs:
- [x] #51351 <!-- Remove boxing in pinv -->
- [x] #52678 <!-- Profile: Improve module docstring -->
- [x] #54201 <!-- Fix generic triangular solves with empty matrices -->
- [x] #54605 <!-- Allow libquadmath to also fail as it is not available
on all systems -->
- [x] #54634 <!-- Fix trampoline assembly for build on clang 18 on apple
silicon -->
- [x] #54635 <!-- Aggressive constprop in trevc! to stabilize triangular
eigvec -->
- [x] #54645 <!-- ensure we set the right value to gc_first_tid -->
- [x] #54671 <!-- Add boundscheck in bindingkey_eq to avoid OOB access
due to data race -->
- [x] #54672 <!-- make: Fix `sed` command for LLVM libraries with no
symbol versioning -->
- [x] #54704 <!-- LazyString in reinterpretarray error messages -->
- [x] #54713 <!-- make: use `readelf` for LLVM symbol version detection
-->
- [x] #54781 <!-- [LinearAlgebra] Improve resilience to unknown
libblastrampoline flags -->
- [x] #54837 <!-- Do not add type tag size to the `alloc_typed` lowering
for GC allocations -->
- [x] #54815 <!-- add sticky task warning to `@task` and `schedule` -->
- [x] #55141 <!-- Update the aarch64 devdocs to reflect the current
state of its support -->
- [x] #55178 <!-- Compat for `Base.@nospecializeinfer` -->
- [x] #55197 <!-- compat notice for a[begin] indexing -->
- [x] #55209 <!-- correction to compat notice for a[begin] -->
- [x] #55203 <!-- document mutable struct const fields -->
- [x] #54769 <!-- add missing compat entry to edit -->
- [x] #54791 <!-- Bump libblastrampoline to v5.10.1 -->
- [x] #55070 <!-- LinearAlgebra: LazyString in error messages for
Diagonal/Bidiagonal -->
- [x] #54624 <!-- more precise aliasing checks for SubArray -->
- [x] #54690 <!-- Fix assertion/crash when optimizing function with dead
basic block -->
- [x] #55084 <!-- Use triple quotes in TOML.print when string contains
newline -->


Need manual backport:
- [ ] #52505 <!-- fix alignment of emit_unbox_store copy -->
- [ ] #53373 <!-- fix sysimage-native-code=no option with pkgimages -->
- [ ] #53984 <!-- Profile: fix heap snapshot is valid char check -->
- [ ] #54276 <!-- Fix solve for complex `Hermitian` with non-vanishing
imaginary part on diagonal -->
- [ ] #54669 <!-- Improve error message in inplace transpose -->
- [ ] #54871 <!-- Make warn missed transformations pass optional -->

Contains multiple commits, manual intervention needed:
- [ ] #52854 <!-- Change to streaming out the heap snapshot data -->
- [ ] #53218 <!-- Fix interpreter_exec.jl test -->
- [ ] #53833 <!-- Profile: make heap snapshots viewable in vscode viewer
-->
- [ ] #54303 <!-- LinearAlgebra: improve type-inference in
Symmetric/Hermitian matmul -->
- [ ] #52694 <!-- Reinstate similar for AbstractQ for backward
compatibility -->
- [ ] #54737 <!-- LazyString in interpolated error messages involving
types -->
- [ ] #54738 <!-- serialization: fix relocatability bug -->
- [ ] #55052 <!-- Fix `(l/r)mul!` with `Diagonal`/`Bidiagonal` -->

Non-merged PRs with backport label:
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #55013 <!-- [docs] change docstring to match code -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
- [ ] #41244 <!-- Fix shell `cd` error when working dir has been deleted
-->
KristofferC added a commit that referenced this pull request Aug 26, 2024
Backported PRs:
- [x] #54962 <!-- Add timing to precompile trace compile -->
- [x] #55180 <!-- compress jit debuginfo for easy memory savings -->
- [x] #54919 <!-- Fix annotated join with non-concrete eltype iters -->
- [x] #55013 <!-- [docs] change docstring to match code -->
- [x] #55017 <!-- TOML: Make `Dates` a type parameter -->
- [x] #54033 <!-- Fix a bug in `stack`'s DimensionMismatch error message
-->
- [x] #55242 <!-- fix at-main docstring to not code quote a compat box
-->
- [x] #55261 <!-- Make `jl_*affinity` tests more portable -->
- [x] #54736 <!-- specificity: ensure fast-path in `sub/eq_msp` handle
missing `UnionAll` wrapper correctly. -->
- [x] #55299 <!-- typeintersect: fix bounds merging during inner
`intersect_all`. -->
- [x] #55302 <!-- Add `lbt_forwarded_funcs()` to debug LBT forwarding
issues -->
- [x] #55148 <!-- Random: Mark unexported public symbols as public -->
- [x] #55303 <!-- avoid overflowing show for OffsetArrays around typemax
-->
- [x] #55317 <!-- Restrict argument to `isleapyear(::Integer)` -->
- [x] #55327 <!-- Profile: Fix stdlib paths -->
- [x] #55330 <!-- [libblastrampoline] Bump to v5.11.0 -->
- [x] #55310 <!-- Preserve structure in scaling triangular matrices by
NaN -->
- [x] #55329 <!-- mapreduce: don't inbounds unknown functions -->
- [x] #55356 <!-- Profile: close files when assembling heap snapshot -->
- [x] #55371 <!-- Fix tr for block SymTridiagonal -->
- [x] #55307 <!-- Make REPL.TerminalMenus public -->
- [x] #55362 <!-- inference: fix missing LimitedAccuracy markers -->
- [x] #55306 <!-- AllocOpt: Fix stack lowering where alloca continas
boxed and unboxed data -->
- [x] #55395 <!-- fix #55389: type-unstable `join` -->
- [x] #55226 <!-- re-add `unsafe_convert` for Reinterpret and Reshaped
array -->
- [x] #55405 <!-- handle unbound vars in NTuple fields -->
- [x] #55365 <!-- ml-matches: ensure all methods are included -->
- [x] #55428 <!-- codegen: move undef freeze before promotion point -->
- [x] #55419 <!-- `stale_cachefile`: handle if the expected cache file
is missing -->
- [x] #55470 <!-- Add push! implementation for AbstractArray depending
only on resize! -->
- [x] #55483 <!-- fix hierarchy level of "API reference" in `Dates`
documentation -->
- [x] #55268 <!-- simplify complex atanh and remove singularity
perturbation -->
- [x] #55441 <!-- fix Event to use normal Condition variable -->
- [x] #55413 <!-- subtyping: fast path for lhs union and rhs typevar -->
- [x] #55492 <!-- build: add missing dependencies for expmap -->
- [x] #55507 <!-- Fix fast getptls ccall lowering. -->
- [x] #55424 <!-- add missing clamp function for IOBuffer -->
- [x] #55504 <!-- Update symmetric docstring to reflect the type of uplo
-->
- [x] #55107 <!-- Make the memory GEP an inbounds GEP since the bounds
check has happened somewhere else -->
- [x] #55411 <!-- Vendor the terminfo database for use with
base/terminfo.jl -->
- [x] #55452 <!-- Do not load `ScopedValues` with `using` -->
- [x] #55407 <!-- Remove deprecated non string API for LLVM pass
pipeline and parse all options -->
- [x] #55461 <!-- 🤖 [master] Bump the StyledStrings stdlib from d7496d2
to f6035eb -->
- [x] #55433 <!-- Backport #55407
to 1.11 -->
- [x] #55225 <!-- [1.11 backport] trace-compile: don't generate
`precompile` statements for OpaqueClosure methods (#55072) -->
- [x] #55212 <!-- Make `Base.depwarn()` public -->
- [x] #552
- [x] #55052 <!-- Fix `(l/r)mul!` with `Diagonal`/`Bidiagonal` -->
- [x] #55251 <!-- Restrict binary ops for Diagonal and Symmetric to
Number eltypes -->95 <!-- LAPACK: Aggressive constprop to concretely
infer syev!/syevd! -->
- [x] #55522 <!-- Fix tr for Symmetric/Hermitian block matrices -->

Need manual backport:
- [x] #55342 <!-- Ensure bidiagonal setindex! does not read indices in
error message -->

Contains multiple commits, manual intervention needed:

- [ ] #55336 <!-- codegen: take gc roots (and alloca alignment) more
seriously -->


Non-merged PRs with backport label:
- [ ] #55506 <!-- Fix indexing in _mapreducedim for OffsetArrays -->
- [ ] #55500 <!-- make jl_thread_suspend_and_get_state safe -->
- [ ] #55499 <!-- propagate the terminal's `displaysize` to the
`IOContext` used by the REPL -->
- [ ] #55458 <!-- Allow for generically extracting unannotated string
-->
- [ ] #55457 <!-- Make AnnotateChar equality consider annotations -->
- [ ] #55453 <!-- Privatise the annotations API, for StyledStrings -->
- [ ] #55443 <!-- Add test for upper/lower/titlecase and fix call -->
- [ ] #55355 <!-- relocation: account for trailing path separator in
depot paths -->
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
- [ ] #41244 <!-- Fix shell `cd` error when working dir has been deleted
-->
@KristofferC KristofferC mentioned this pull request Aug 26, 2024
33 tasks
@KristofferC KristofferC mentioned this pull request Sep 12, 2024
63 tasks
KristofferC added a commit that referenced this pull request Sep 17, 2024
Backported PRs:
- [x] #55480 <!-- Fix push! for OffsetVectors, add tests for push! and
append! on AbstractVector -->
- [x] #55443 <!-- Add test for upper/lower/titlecase and fix call -->
- [x] #55524 <!-- Set `.jl` sources as read-only during installation -->
- [x] #55500 <!-- make jl_thread_suspend_and_get_state safe -->
- [x] #55506 <!-- Fix indexing in _mapreducedim for OffsetArrays -->
- [x] #55564 <!-- Empty out loaded_precompiles dict instead of asserting
it's empty. -->
- [x] #55567 <!-- Initialize threadpools correctly during sysimg build
-->
- [x] #55596 <!-- Fast bounds-check for CartesianIndex ranges -->
- [x] #55605 <!-- Reroute Symmetric/Hermitian + Diagonal through
triangular -->
- [x] #55640 <!-- win: move stack_overflow_warning to the backtrace
fiber -->
- [x] #55715 <!-- Add precompile signatures to Markdown to reduce
latency. -->
- [x] #55593 <!-- Fix invalidations for FileIO -->
- [x] #55555 <!-- Revert "Don't expose guard pages to malloc_stack API
consumers" -->
- [x] #55720 <!-- Fix `pkgdir` for extensions -->
- [x] #55729 <!-- Avoid confounding compilation side effects of
`@time_imports` -->
- [x] #55718 <!-- Fix `@time_imports` extension recognition -->
- [x] #55522 <!-- Fix tr for Symmetric/Hermitian block matrices -->

Contains multiple commits, manual intervention needed:
- [ ] #55509 <!-- Fix cong implementation to be properly random and not
just cycling. -->

Non-merged PRs with backport label:
- [ ] #55641 <!-- fall back to slower stat filesize if optimized
filesize fails -->
- [ ] #55534 <!-- Set stdlib sources as read-only during installation
-->
- [ ] #55499 <!-- propagate the terminal's `displaysize` to the
`IOContext` used by the REPL -->
- [ ] #55458 <!-- Allow for generically extracting unannotated string
-->
- [ ] #55457 <!-- Make AnnotateChar equality consider annotations -->
- [ ] #55453 <!-- Privatise the annotations API, for StyledStrings -->
- [ ] #55355 <!-- relocation: account for trailing path separator in
depot paths -->
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
- [ ] #41244 <!-- Fix shell `cd` error when working dir has been deleted
-->
@KristofferC KristofferC mentioned this pull request Sep 24, 2024
30 tasks
KristofferC added a commit that referenced this pull request Sep 25, 2024
Backported PRs:
- [x] #55773 <!-- Add compat entry for `Base.donotdelete` -->
- [x] #41244 <!-- Fix shell `cd` error when working dir has been deleted
-->
- [x] #55795 <!-- fix #52986, regression in `@doc` of macro without REPL
loaded -->
- [x] #55829 <!-- [Dates] Make test more robust against non-UTC
timezones -->
- [x] #55641 <!-- fall back to slower stat filesize if optimized
filesize fails -->
- [x] #55744 <!-- fix #45494, error in ssa conversion with complex type
decl -->
- [x] #55783 <!-- use `inferencebarrier` instead of `invokelatest` for
1-arg `@assert` -->
- [x] #55739 <!-- Add `invokelatest` barrier to `string(...)` in
`@assert` -->

Need manual backport:
- [ ] #55798 <!-- Broadcast binary ops involving strided triangular -->

Contains multiple commits, manual intervention needed:
- [ ] #55509 <!-- Fix cong implementation to be properly random and not
just cycling. -->
- [ ] #55569 <!-- Add a docs section about loading/precomp/ttfx time
tuning -->
- [ ] #55824 <!-- Replace regex package module checks with actual code
checks -->

Non-merged PRs with backport label:
- [ ] #55845 <!-- privatize annotated string API, take two -->
- [ ] #55828 <!-- Fix some corner cases of `isapprox` with unsigned
integers -->
- [ ] #55813 <!-- Check for conflicting `@ccallable` name before JIT
registration -->
- [ ] #55743 <!-- doc: heap snapshot viewing -->
- [ ] #55741 <!-- Change annotations to use a NamedTuple -->
- [ ] #55534 <!-- Set stdlib sources as read-only during installation
-->
- [ ] #55499 <!-- propagate the terminal's `displaysize` to the
`IOContext` used by the REPL -->
- [ ] #55458 <!-- Allow for generically extracting unannotated string
-->
- [ ] #55457 <!-- Make AnnotateChar equality consider annotations -->
- [ ] #55453 <!-- Privatise the annotations API, for StyledStrings -->
- [ ] #55355 <!-- relocation: account for trailing path separator in
depot paths -->
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
@KristofferC KristofferC mentioned this pull request Sep 30, 2024
39 tasks
KristofferC added a commit that referenced this pull request Oct 1, 2024
Backported PRs:
- [x] #55849 <!-- Mmap: fix grow! for non file IOs -->
- [x] #55863 <!-- Update TaskLocalRNG docstring according to #49110 -->
- [x] #54433 <!-- Root globals in toplevel exprs -->
- [x] #55828 <!-- Fix some corner cases of `isapprox` with unsigned
integers -->
- [x] #55890 <!-- Profile: fix order of fields in heapsnapshot & improve
formatting -->
- [x] #55884 <!-- inference: add missing `TypeVar` handling for
`instanceof_tfunc` -->
- [x] #55881 <!-- Install terminfo data under /usr/share/julia -->
- [x] #55909 <!-- do not intentionally suppress errors in precompile
script from being reported or failing the result -->
- [x] #55355 <!-- relocation: account for trailing path separator in
depot paths -->
- [x] #55917 <!-- fix rawbigints OOB issues -->
- [x] #55892 <!-- TOML: Avoid type-pirating `Base.TOML.Parser` -->
- [x] #55798 <!-- Broadcast binary ops involving strided triangular -->
- [x] #55919 <!-- Limit `@inbounds` to indexing in the dual-iterator
branch in `copyto_unaliased!` -->

Contains multiple commits, manual intervention needed:
- [ ] #54009 <!-- allow extensions to trigger from packages in [deps]
-->
- [ ] #55509 <!-- Fix cong implementation to be properly random and not
just cycling. -->
- [ ] #55569 <!-- Add a docs section about loading/precomp/ttfx time
tuning -->
- [ ] #55824 <!-- Replace regex package module checks with actual code
checks -->

Non-merged PRs with backport label:
- [ ] #55932 <!-- REPL: make UndefVarError aware of imported modules -->
- [ ] #55910 <!-- Prevent extensions from blocking parallel
pre-compilation -->
- [ ] #55908 <!-- add logic to prefer loading modules that are already
loaded -->
- [ ] #55886 <!-- irrationals: restrict assume effects annotations to
known types -->
- [ ] #55871 <!-- lowering: don't reverse handler order in
`(pop-handler-list ...)` -->
- [ ] #55870 <!-- fix infinite recursion in `promote_type` for
`Irrational` -->
- [ ] #55867 <!-- update `hash` doc string: `widen` not required any
more -->
- [ ] #55851 <!-- [REPL] Fix #55850 by using `safe_realpath` instead of
`abspath` in `projname` -->
- [ ] #55813 <!-- Check for conflicting `@ccallable` name before JIT
registration -->
- [ ] #55743 <!-- doc: heap snapshot viewing -->
- [ ] #55741 <!-- Change annotations to use a NamedTuple -->
- [ ] #55534 <!-- Set stdlib sources as read-only during installation
-->
- [ ] #55499 <!-- propagate the terminal's `displaysize` to the
`IOContext` used by the REPL -->
- [ ] #55458 <!-- Allow for generically extracting unannotated string
-->
- [ ] #55457 <!-- Make AnnotateChar equality consider annotations -->
- [ ] #55220 <!-- `isfile_casesensitive` fixes on Windows -->
- [ ] #55169 <!-- `propertynames` for SVD respects private argument -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
- [ ] #50813 <!-- More doctests for Sockets and capitalization fix -->
- [ ] #50157 <!-- improve docs for `@inbounds` and
`Base.@propagate_inbounds` -->
@KristofferC KristofferC mentioned this pull request Oct 7, 2024
44 tasks
@KristofferC KristofferC mentioned this pull request Oct 18, 2024
43 tasks
@KristofferC KristofferC mentioned this pull request Oct 29, 2024
47 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 1.10 Change should be backported to the 1.10 release backport 1.11 Change should be backported to release-1.11
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Problem with isfile_casesensitive() on Windows
2 participants