Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error while training with legged_gym #6

Open
mihirk284 opened this issue Jun 7, 2022 · 3 comments
Open

Error while training with legged_gym #6

mihirk284 opened this issue Jun 7, 2022 · 3 comments

Comments

@mihirk284
Copy link

I have installed the legged_gym and am trying to run the default task of the ANYmal C robot on flat terrain.

However, after a few seconds of simulation (The simulator screen appears and the robots show some actions being applied to them), the following error is encountered. I am not sure how to solve this, however, it seems that this is caused by the rsl_rl package rather than the legged_gym package.

I am attaching the output below. Kindly advise on how to resolve this.

Importing module 'gym_37' (/home/arl/Documents/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/arl/Documents/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.8.1
Device count 2
/home/arl/Documents/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/arl/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/arl/.cache/torch_extensions/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
Setting seed: 1
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
Actor MLP: Sequential(
  (0): Linear(in_features=48, out_features=128, bias=True)
  (1): ELU(alpha=1.0)
  (2): Linear(in_features=128, out_features=64, bias=True)
  (3): ELU(alpha=1.0)
  (4): Linear(in_features=64, out_features=32, bias=True)
  (5): ELU(alpha=1.0)
  (6): Linear(in_features=32, out_features=12, bias=True)
)
Critic MLP: Sequential(
  (0): Linear(in_features=48, out_features=128, bias=True)
  (1): ELU(alpha=1.0)
  (2): Linear(in_features=128, out_features=64, bias=True)
  (3): ELU(alpha=1.0)
  (4): Linear(in_features=64, out_features=32, bias=True)
  (5): ELU(alpha=1.0)
  (6): Linear(in_features=32, out_features=1, bias=True)
)
/home/arl/miniconda3/envs/rlgpu/lib/python3.7/site-packages/torch/nn/modules/module.py:889: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters(). (Triggered internally at  /opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cudnn/RNN.cpp:915.)
  result = self.forward(*input, **kwargs)
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [8,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [9,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [10,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [11,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [13,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [16,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [17,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [28,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [29,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [30,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [31,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [40,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [108,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [100,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [101,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [102,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [103,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [104,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [105,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [106,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [107,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [108,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [109,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [110,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [111,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [112,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [113,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [114,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [115,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [116,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [117,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [118,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [119,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [120,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [122,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [123,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [124,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [125,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [126,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [94,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [46,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [47,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [48,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [49,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [50,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [51,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [52,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [53,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [54,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [55,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [56,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [57,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [58,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [59,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [60,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [61,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [62,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [150,0,0], thread: [63,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [8,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [9,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [10,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [11,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [13,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [16,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [17,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [18,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [19,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [28,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [29,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [30,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [31,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [64,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [65,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [66,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [67,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [68,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [69,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [70,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [71,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [72,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [73,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [74,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [75,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [76,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [77,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [78,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [79,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [80,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [81,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [82,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [83,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [84,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [85,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [86,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [87,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [88,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [89,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [90,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [91,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [92,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [93,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [94,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1616554800319/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [136,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "legged_gym/scripts/train.py", line 47, in <module>
    train(args)
  File "legged_gym/scripts/train.py", line 43, in train
    ppo_runner.learn(num_learning_iterations=train_cfg.runner.max_iterations, init_at_random_ep_len=True)
  File "/home/arl/Documents/rsl_rl/rsl_rl/runners/on_policy_runner.py", line 132, in learn
    mean_value_loss, mean_surrogate_loss = self.alg.update()
  File "/home/arl/Documents/rsl_rl/rsl_rl/algorithms/ppo.py", line 131, in update
    self.actor_critic.act(obs_batch, masks=masks_batch, hidden_states=hid_states_batch[0])
  File "/home/arl/Documents/rsl_rl/rsl_rl/modules/actor_critic.py", line 124, in act
    self.update_distribution(observations)
  File "/home/arl/Documents/rsl_rl/rsl_rl/modules/actor_critic.py", line 121, in update_distribution
    self.distribution = Normal(mean, mean*0. + self.std)
  File "/home/arl/miniconda3/envs/rlgpu/lib/python3.7/site-packages/torch/distributions/normal.py", line 50, in __init__
    super(Normal, self).__init__(batch_shape, validate_args=validate_args)
  File "/home/arl/miniconda3/envs/rlgpu/lib/python3.7/site-packages/torch/distributions/distribution.py", line 52, in __init__
    if not constraint.check(getattr(self, param)).all():
RuntimeError: CUDA error: device-side assert triggered
@Parth1811
Copy link

DId you find a solution to this problem?

@Parth1811
Copy link

One solution that worked for me was to use --rl_device=cpu.
Was trying to replicate the error using CPU to get a good Traceback but it just ran without errors

@mihirk284
Copy link
Author

Hi,
I did not find a solution to this problem and also did not spend a lot of time on this.

However, this repository has been significantly updated since I opened this issue and hopefully this might have been resolved in this time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Parth1811 @mihirk284