../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)

[Darshcg]

Hi @r7vme,

I have Implemented Two Custom Plugins(Einsum and RoIAlign). For FP32 it is working fine during ONNX to TensorRT Conversion using the command: ./trtexec --onnx=slow_fast_1.onnx --shapes=input1:1x3x8x256x455,input2:1x3x32x256x455,input3:1x5 --plugins=Einsum_op.so --plugins=RoI_Align.so --saveEngine=slow_fast_FP32.trt.

For FP16 I am getting an issue as: ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)

Not sure what exactly is going wrong.

Can you please help me to resolve this Issue?

I just changed the CUBLAS Implementation of Einsum from cublasSgemmStridedBatched(FP32) to cublasHgemmStridedBatched(FP16).

FP32 Cublas Implementation:

cublasSgemmStridedBatched(mCublas, CUBLAS_OP_N,CUBLAS_OP_T, 
                                  M, K,L,&onef, 
                                  reinterpret_cast<const float *>(inputs[1]), M, 1,
                                  reinterpret_cast<const float *>(inputs[0]), K, 1,
                                  &zerof, 
                                  reinterpret_cast<float *>(outputs[0]), 
                                  M,1, N);

FP16 Cublas Implementation:

cublasHgemmStridedBatched(mCublas, CUBLAS_OP_N,CUBLAS_OP_T, 
                                  M, K,L,&oneh, 
                                  reinterpret_cast<const __half *>(inputs[1]), M, 1,
                                  reinterpret_cast<const __half *>(inputs[0]), K, 1,
                                  &zeroh, 
                                  reinterpret_cast<__half *>(outputs[0]), 
                                  M,1, N);

Below is the error Log which I am getting:

[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during timeReformat.
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during timeReformat.
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Div_494
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during timeReformat.
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during timeReformat.
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Einsum_495

[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during timeReformat.
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during timeReformat.
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Conv_811 + Add_813 + Relu_814
[03/05/2021-15:28:12] [E] [TRT] ../rtSafe/safeRuntime.cpp (25) - Cuda Error in allocate: 2 (out of memory)
[03/05/2021-15:28:12] [W] [TRT] GPU memory allocation error during getBestTactic: Conv_811 + Add_813 + Relu_814
[03/05/2021-15:28:12] [E] [TRT] Try increasing the workspace size with IBuilderConfig::setMaxWorkspaceSize() if using IBuilder::buildEngineWithConfig, or IBuilder::setMaxWorkspaceSize() if using IBuilder::buildCudaEngine.
[03/05/2021-15:28:12] [E] [TRT] ../builder/tacticOptimizer.cpp (1715) - TRTInternal Error in computeCosts: 0 (Could not find any implementation for node Conv_811 + Add_813 + Relu_814.)

May I know what exactly is the issue? Can you please assist me to resolve this.

Thanks,
Darshan

[bodhi-tree01]

I'm having the same problem as you, and I haven't found a solution yet. Frustration.