Some package load times affected drastically by current working directory #36911
Labels
system:windows
Affects only Windows
Comments
|
I also found a really strong correlation between slow disk io speed and long package load times in #35907 It would be great to get to the bottom of why julia diskio can be really slow in non-linux systems. Even linux VMs on MacOS systems are a lot faster on disk io than in the parent MacOS env. i.e. the same hardware Also, could code loading read/write to disk less frequently? Might that help? |
|
Maybe try #36018. |
|
Is the package loading code looking into the current working directory? In my case, all the code that needs to be loaded is on a local fast SSD drive. On the local drive On the remote |
|
Testing nightly With and when With Julia prompt on SSD With Julia prompt on a network share Any suggestions to proceed, or what additional information would be useful? |
|
Maybe |
|
Some handholding needed :) |
|
There's nothing there consistent with a big difference in execution times. I'm puzzled, but it's hard to know what to suggest "through the ether" other than to try to come to some conclusion about whether You can hover over bars to see where the time that it does record is coming from, so as a precaution check that. |
|
Re-testing with Netshare
Local disk
|
|
Are you getting any warnings about the profile buffer overflowing? |
No warnings. |
|
I think I saw a note somewhere that the warnings are busted. Do |
Netshare
Local disk
|
|
Great! Now hover over bars along the top that take more horizontal space. You can click on them and the source line will be printed at the REPL. What are the main culprits for the "slow" case? |
|
Starting from top, trying to click the longest ones, and going a bit down gives |
|
For my education, this is the slow case annotated |
|
Maybe a simpler example. First time a non-existing library is tried to be opened, on the slow netshare it takes 36 s, but subsequent Netshare
Local disk
This is Julia v"1.6.0-DEV.1210" on Windows 10. Is there an easy way to make |
|
You can use sysinternals procmon to look at the file system accesses
invoked by Julia. That’s probably the closest thing to what you want.
-E
…On Wed, Oct 14, 2020 at 08:03 jaakkor2 ***@***.***> wrote:
Maybe a simpler example. First time a non-existing library is tried to be
opened, on the slow netshare it takes 36 s, but subsequent dlopen on a
non-existing library is fast.
Netshare
@time stat(".") 1.368621 seconds (1 allocation: 224 bytes)
julia> using Dates
julia> t1=now(); try @time Base.Libc.Libdl.dlopen(joinpath(Sys.BINDIR, "libwinpthread-1.dll")); catch; println("not found"); end; now()-t1
0.000515 seconds (17 allocations: 1.062 KiB)
65 milliseconds
julia> t1=now(); try @time Base.Libc.Libdl.dlopen(joinpath(Sys.BINDIR, "libwinpthread-2.dll")); catch; println("not found"); end; now()-t1
not found
36433 milliseconds
julia> t1=now(); try @time Base.Libc.Libdl.dlopen(joinpath(Sys.BINDIR, "libwinpthread-3.dll")); catch; println("not found"); end; now()-t1
not found
0 milliseconds
Local disk
@time stat(".") 0.000165 seconds (1 allocation: 224 bytes)
julia> using Dates
julia> t1=now(); try @time Base.Libc.Libdl.dlopen(joinpath(Sys.BINDIR, "libwinpthread-1.dll")); catch; println("not found"); end; now()-t1
0.000870 seconds (17 allocations: 1.062 KiB)
65 milliseconds
julia> t1=now(); try @time Base.Libc.Libdl.dlopen(joinpath(Sys.BINDIR, "libwinpthread-2.dll")); catch; println("not found"); end; now()-t1
not found
182 milliseconds
julia> t1=now(); try @time Base.Libc.Libdl.dlopen(joinpath(Sys.BINDIR, "libwinpthread-3.dll")); catch; println("not found"); end; now()-t1
not found
0 milliseconds
This is Julia v"1.6.0-DEV.1210" on Windows 10.
Is there an easy way to make dlopen to printout the library names it is
trying to open?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#36911 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAA762GBH4EOJZ7DTMAFOR3SKW4TTANCNFSM4PUHQIEA>
.
|
|
https://docs.microsoft.com/en-us/sysinternals/downloads/procmon seems to do the job. creates 737 events with a path on |
|
Fantastic; this is great info. @vtjnash I assume this is due to Windows searching for debugging symbols, and doing so in the current directory by default. Is there any way we can suppress that behavior? |
|
Yes, it's one of the arguments we [don't] pass to SymInitialize |
|
Cool, thanks. @jaakkor2 if you run this piece of code first, does it speed things up at all: |
|
Improves quite a lot, but still not a speed king when the current directory is on a slow netshare. NetshareBefore (I believe After Local diskNew logLogging events gives 161 events, now current directory is still scanned for debug files, but |
|
Hmmm. I could have sworn that we told it not to look there. :P If you are compiling Julia locally, could you also try modifying this line to pass |
|
I was filtering the path to contain
I am not usually compiling locally, maybe in the weekend. If someone wants to try the modification in #36911 (comment), here are the commands and monitor with https://docs.microsoft.com/en-us/sysinternals/downloads/procmon that preferably |
|
Calling NetshareBefore After Local diskBefore After If I call I noticed that on v1.5.2 debug files are scanned in the current working directory during the start up, but in v.1.6.0-DEV.1210 not. This explains that on v1.6.0 first |
|
Setting the
The benefit of setting Effect on |
|
Finally, this works i.e., set the SymSetOptions docs say this about
|
vtjnash
pushed a commit
that referenced
this issue
Oct 20, 2020
… Windows (#38085) Fixes #36911 [SymSetOptions](https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symsetoptions) docs say this about SYMOPT_IGNORE_CVREC: > Ignore path information in the CodeView record of the image header when loading a .pdb file.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I noticed that if I change directory to a slow network share, some package load times become up to 45x longer.
Loading Distributions.jl is reasonably fast
but if i change directory to a relatively slow network share, loading becomes much slower
My
startup.jlwas empty (therefore not an issue with Revise.jl), and the network share folder is empty as well.This is on Windows 10, Julia v1.5.0, but v1.4.1 shows similar behavior.
Not only Distributions.jl, but
gives
with the three repeats with restarted Julia. The slowest one was with the highest network load, but
The text was updated successfully, but these errors were encountered: