SdfLayer::~SdfLayer deadlock in Python

[nvmkuruc]

Description of Issue

SdfLayer::~SdfLayer appears to deadlock in Python. When running parallel tasks operating on layers opened via Sdf.Layer.FindOrOpen, we observe deadlocks. The layers are not opened via Usd.Stage or retained beyond the lifetime of their task.

Using TF_DEBUG=SDF_* suggests a deadlock in SdfLayer::~SdfLayer(). Other SdfLayer operations release the GIL before acquiring the layer registry lock, noting the potential for deadlocks.

Adding TF_PY_ALLOW_THREADS_IN_SCOPE(); right after the TF_DEBUG statement at the top of ~SdfLayer() appears to avoid the deadlock.

Steps to Reproduce

Run testSdfLayer with the additional test case added below. It usually reproduces with 10 iterations, but you may need more or to run the test multiple times. It has not reproduced on Linux, only Windows.

    def test_OpenCloseThreadSafety(self):
        PATH = "thread_safety.sdf"
        OPENS = 25
        WORKERS = 16
        ITERATIONS = 10
        layer = Sdf.Layer.CreateNew(PATH)
        layer.Save()
        del layer

        for _ in range(ITERATIONS):
            with (concurrent.futures.ThreadPoolExecutor(max_workers=WORKERS)
                as executor):
                futures = [executor.submit(
                            lambda: bool(Sdf.Layer.FindOrOpen(PATH))
                           ) for _ in range(OPENS)]
                completed = sum(1 if f.result() else 0 for f in
                                concurrent.futures.as_completed(futures))
            self.assertEqual(completed, OPENS)

The deadlock does not reproduce when the GIL is released in ~SdfLayer().

System Information (OS, Hardware)

Windows 10

Package Versions

Python 3.9

Build Flags

[tallytalwar]

Filed as internal issue #USD-8046