Windows file read coherence

[bfloch]

Description of Issue

We have network mounts to our USD files.
When a USD file "A" is open in any DCC (!) replacing the underlying file will have no effect for any other USD files.
We replace file "A" with "B" on the server, but opening it in any other DCC will still open it as "A" as long as some process is holding on to it.

Reload Stage will have no effect either.

This was the weirdest behaviour since we could not see it with any other files, applications.

I am not a Windows Guru, but considering that by default a mapped file is used for loading - a mechanism that at least in Windows is mainly documented for sharing memory between processes, we wonder if that could have to do with it.

We tried exporting USDC_USE_PREAD=1 with no effect.

Note that when another thirdparty software, like powershells Get-FileHash reads from the file, not only does it get the right result,
but in this case it also corrupts the open USD Sessions.

After that, or when all USD processes close, the file open "B" as expected again.

This is a somewhat critical problem for us - at it breaks any expectations we might have on working within DCCs.
I hope this is reproducible for others - else we might need to look closer into own configurations or invest into techniques around the resolver instead.

This might be related to:
#108
#1377

Steps to Reproduce

1. Open an USD File on a shared network (e.g. usdview)
2. Replace the File on the server with something else
3. Open the replaced USD File in a separate process (e.g. another usdview) - it will show as the original not as the new content

Either close both files to flush it - or access the file via a separate app like Get-FileHash. In the latter case the USD state
is corrupted in the running DCCs. ReloadStage will crash for us.

System Information (OS, Hardware)

Windows 10

Package Versions

Any

Build Flags

Default

[spiffmon]

Hi @bfloch ,
Sorry to hear you're having issues! We don't have alot of Windows expertise, either, and about all I can offer at the moment is that it is a legitimate, performant practice to read data out of files by memory mapping them, thus bypassing the extra buffering that comes with FILE / stream IO.
And in fact, if you have USDC_USE_PREAD=1 in your DCC's environments, then we can rule out mmapping as the problem.
Network filesystems are always tricky when it comes to multi-client interleaved reading and writing, and the thing that I suspect makes USD seem more fragile is that we keep the binary files open effectively till you close all stages that use the file. Which does make me suspicious that you say you still get a crash if you close the stage in a process after swapping out a file, and then reopening it... it makes me suspect that a UsdStageCache (or some other cache) is keeping the stage alive. You can set the env var TF_DEBUG=USD_STAGE_* to see how stages are actually being created and destroyed (or not).

We use the "Create to temporary file and atomic move into place" pattern for editing/replacing files that may be open in other apps, and have not (as far as I'm aware) experienced the kind of crashes you're seeing on Windows, and in fact, once the new file is in place, a stage->Reload() should pick up the new data.
The two other suggestions I can make are:

1. Do you get different/better behavior when all the processes and the files are local to the same machine?
2. You might try posting about this behavior on usd-interest, as there could be more folks watching there than here on GitHub.

[bfloch]

Thanks Sebastian.

This is very helpful information and helped me pinpoint the issue further down within our oplock configuration of our windows facing fileserver. I was able to replicate an incoherent state with simple files, and while it is still interesting why a reload of a stage would fail/crash - I don't have enough information yet to verify that the behaviour is specific to USD.

I don't think I have a real case against USD's codebase here, so I'll close the ticket and forward further question towards the mailing list, as suggested.