realpath() and macOS Catalina

[cormullion]

On last year's macOS (Mojave), I can do this:

cd(homedir() * "/.julia/registries/General")
map(realpath, readdir())

and get a 31-element Array{String,1}:.

On this year's macOS (Catalina), it goes wrong:

ERROR: SystemError: realpath: No such file or directory
Stacktrace:
 [1] #systemerror#44(::Nothing, ::typeof(systemerror), ::Symbol, ::Bool) at ./error.jl:134
 [2] systemerror at ./error.jl:134 [inlined]
 [3] realpath(::String) at ./path.jl:380
 [4] iterate at ./generator.jl:47 [inlined]
 [5] collect_to! at ./array.jl:651 [inlined]
 [6] collect_to_with_first!(::Array{String,1}, ::String, ::Base.Generator{Array{String,1},typeof(realpath)}, ::Int64) at ./array.jl:630
 [7] _collect(::Array{String,1}, ::Base.Generator{Array{String,1},typeof(realpath)}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:624
 [8] collect_similar(::Array{String,1}, ::Base.Generator{Array{String,1},typeof(realpath)}) at ./array.jl:548
 [9] map(::Function, ::Array{String,1}) at ./abstractarray.jl:2073
 [10] top-level scope at none:0

I'm curious if anyone else can reproduce this, or is it just a problem for me... :)

Edit: Julia 1.2. It’s fine on version 1.1.

[Nosferican]

Works fine for me.

[dkarrasch]

Can reproduce on Julia v1.2, but works fine on v1.3 and a 5-day-old nightly.

[cormullion]

i ran the Julia 1.2 tests - the ones with realpath all failed with similar errors. Eg

Error in testset file:
Error During Test at /Applications/Julia-1.2.app/Contents/Resources/julia/share/julia/test/file.jl:1051
  Test threw exception
  Expression: basename(realpath(uppercase(path))) == path
  SystemError: realpath: No such file or directory
  Stacktrace:
   [1] #systemerror#44(::Nothing, ::typeof(systemerror), ::Symbol, ::Bool) at ./error.jl:134
``

[cormullion]

I think this is caused by the new security features introduced in Catatonia. If you open System Preferences>Privacy>Full Disk Access, and add the Julia application to the list of ‘approved’ apps, these start to work again. Might be wrong... 😀

[mkschleg]

Adding Julia app and the bin to the approved apps list seems to work, but it seems like this issue also persists when re-running this function multiple times in a row really quickly.

julia> realpath(".")
"/Users/Matt/Documents/Developer_Projects/JuliaDev/Reproduce.jl"

julia> realpath(".")
"/Users/Matt/Documents/Developer_Projects/JuliaDev/Reproduce.jl"

julia> realpath(".")
"/Users/Matt/Documents/Developer_Projects/JuliaDev/Reproduce.jl"

julia> realpath(".")
ERROR: SystemError: realpath: No such file or directory
Stacktrace:
 [1] #systemerror#44(::Nothing, ::typeof(systemerror), ::Symbol, ::Bool) at ./error.jl:134
 [2] systemerror at ./error.jl:134 [inlined]
 [3] realpath(::String) at ./path.jl:380
 [4] top-level scope at REPL[6]:1

And it still seems really unreliable. Works maybe 1/4 of the time for me, and usually after letting the function "cool down" for a few seconds which is not ideal.

[StefanKarpinski]

I don't know whether to laugh or cry. WTF, Apple?

[mkschleg]

Building from source also doesn't seem to fix it.

I don't know whether to laugh or cry. WTF, Apple?

More like pull my hair out...

[mkschleg]

Also this is all with Julia 1.1.1, but I've had similar issues with 1.2.

[vtjnash]

Maybe this explains why I've heard it being called macOS 10.15 Vista https://twitter.com/tylerhall/status/1181324733893287938

[mkschleg]

Definitely has to do with the new privacy stuff. I'm just happy I have a backup on Mojave and a Linux server to actually run experiments...

[mkschleg]

As with @dkarrasch the 1.3 release candidate works for me. I'm not sure what the difference between 1.1, 1.2, and 1.3. But I had not installed 1.3 before in Mojave, but that would be really odd if it mattered.

[jwscook]

@cormullion's fix works for me. Thanks!

I think this is caused by the new security features introduced in Catatonia. If you open System Preferences>Privacy>Full Disk Access, and add the Julia application to the list of ‘approved’ apps, these start to work again. Might be wrong... 😀

[vtjnash]

This seems to mean that lstat must have failed on some prefix in the path (https://opensource.apple.com/source/Libc/Libc-498/stdlib/FreeBSD/realpath.c.auto.html)

[DilumAluthge]

Maybe backporting #33116 would fix this on Julia 1.2?

[mkschleg]

This function seems to fix the issue in 1.1.1 just from implementing in the repl. So a backport should fix the issue.

[vtjnash]

That's all rather nonsensical then—that commit is mostly NFCI and just removes some platform specific branches by utilizing the ones already in libuv.

[ghost]

@cormullion's fix works for me. Thanks!

I think this is caused by the new security features introduced in Catatonia. If you open System Preferences>Privacy>Full Disk Access, and add the Julia application to the list of ‘approved’ apps, these start to work again. Might be wrong... 😀

Didn't help me.

[vtjnash]

Turns out that "realpath" on macOS gets sometimes renamed to "realpath$DARWIN_EXTSN" (aka _DARWIN_BETTER_REALPATH), so switching to libuv would actually introduce functional changes in behavior.

julia> ccall("realpath\$DARWIN_EXTSN", Cstring, (Cstring, Cstring), "./..", C_NULL)
Cstring(0x00007fedba8c0c00)

julia> unsafe_string(ans)
"/Users/jameson"

$ nm usr/lib/libuv.a | grep realpath
                 U _realpath$DARWIN_EXTSN

Though those changes aren't entirely new (https://github.com/unofficial-opensource-apple/Libc/blob/master/stdlib/FreeBSD/realpath.c from 6 years ago was 10.9).

[vtjnash]

Fixed by #33116