TestNeighbors.py failures with pytorch 1.13

[sef43]

If I install NNPOps with pytorch 13

conda install -c conda-forge nnpops pytorch=1.13

Or build from source with pytorch 1.13 then the tests in TestNeighbors.py fail with a pytorch runtime error, e.g.:

FAILED TestNeighbors.py::test_neighbor_grads[distances-1-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [0]], which is output 0 of NormBackward1, is at version 1;...

Output of

pytest TestNeighbors.py

=================================================================================== short test summary info ====================================================================================
FAILED TestNeighbors.py::test_neighbor_grads[distances-1-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [0]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[distances-1-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [0]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[distances-2-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[distances-2-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [1]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[distances-3-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[distances-3-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [3]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[distances-4-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [6]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[distances-4-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [6]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[distances-5-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[distances-5-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [10]], which is output 0 of NormBackward1, is at version ...
FAILED TestNeighbors.py::test_neighbor_grads[distances-10-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [45]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[distances-10-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [45]], which is output 0 of NormBackward1, is at version ...
FAILED TestNeighbors.py::test_neighbor_grads[distances-100-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4950]], which is output 0 of NormBackward1, is at version...
FAILED TestNeighbors.py::test_neighbor_grads[distances-100-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [4950]], which is output 0 of NormBackward1, is at versio...
FAILED TestNeighbors.py::test_neighbor_grads[distances-1000-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [499500]], which is output 0 of NormBackward1, is at versi...
FAILED TestNeighbors.py::test_neighbor_grads[distances-1000-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [499500]], which is output 0 of NormBackward1, is at vers...
FAILED TestNeighbors.py::test_neighbor_grads[combined-1-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [0]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[combined-1-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [0]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[combined-2-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[combined-2-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [1]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[combined-3-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[combined-3-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [3]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[combined-4-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [6]], which is output 0 of NormBackward1, is at version 1;...
FAILED TestNeighbors.py::test_neighbor_grads[combined-4-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [6]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[combined-5-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[combined-5-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [10]], which is output 0 of NormBackward1, is at version ...
FAILED TestNeighbors.py::test_neighbor_grads[combined-10-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [45]], which is output 0 of NormBackward1, is at version 1...
FAILED TestNeighbors.py::test_neighbor_grads[combined-10-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [45]], which is output 0 of NormBackward1, is at version ...
FAILED TestNeighbors.py::test_neighbor_grads[combined-100-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [4950]], which is output 0 of NormBackward1, is at version...
FAILED TestNeighbors.py::test_neighbor_grads[combined-100-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [4950]], which is output 0 of NormBackward1, is at versio...
FAILED TestNeighbors.py::test_neighbor_grads[combined-1000-dtype0] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [499500]], which is output 0 of NormBackward1, is at versi...
FAILED TestNeighbors.py::test_neighbor_grads[combined-1000-dtype1] - RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor [499500]], which is output 0 of NormBackward1, is at vers...
========================================================================== 32 failed, 214 passed, 2 warnings in 8.07s ==========================================================================

The test passes with pytorch=1.12

Is anyone else able to reproduce?
running on Linux with Cuda 11.7

[RaulPPelaez]

I am compiling from source and I can reproduce this, also using Linux with CUDA 11.7.

[RaulPPelaez]

Torch suggests running the offending code with the following defined:

    pt.autograd.set_detect_anomaly(True)

Running the test with that marks the following lines as offending:

NNPOps/src/pytorch/neighbors/TestNeighbors.py

Line 122 in 16543f9

distances_cpu.sum().backward()

NNPOps/src/pytorch/neighbors/TestNeighbors.py

Line 125 in 16543f9

(deltas_cpu.sum() + distances_cpu.sum()).backward()

Is this meaningful to you? @sef43 @raimis

[sef43]

this only happens for the NNPOps CPU implementation, the changes in #91 seem to fix it