Describe how to check for broken links #1719

mikemckiernan · 2022-11-29T17:43:44Z

This is one way to check for broken links,
but I'm happy to adopt something that is
better.

This is one way to check for broken links, but I'm happy to adopt something that is better.

mikemckiernan · 2022-11-29T17:45:30Z

Let's review and merge after #1711. Several broken links are resolved in that PR.

nvidia-merlin-bot · 2022-11-29T18:08:07Z

Click to view CI Results

GitHub pull request #1719 of commit 0f3608c168e39aa4933c856d12dadc4d3f6016a8, no merge conflicts.
Running as SYSTEM
Setting status of 0f3608c168e39aa4933c856d12dadc4d3f6016a8 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/nvtabular_tests/4908/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
 > git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1719/*:refs/remotes/origin/pr/1719/* # timeout=10
 > git rev-parse 0f3608c168e39aa4933c856d12dadc4d3f6016a8^{commit} # timeout=10
Checking out Revision 0f3608c168e39aa4933c856d12dadc4d3f6016a8 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f3608c168e39aa4933c856d12dadc4d3f6016a8 # timeout=10
Commit message: "Describe how to check for broken links"
 > git rev-list --no-walk 78db85db82def8a1a3ef4a72ece337fd5ed2422d # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins749166207764677682.sh
GLOB sdist-make: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/setup.py
test-gpu create: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/.tmp/package/1/nvtabular-1.6.0+12.g0f3608c1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.3,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,-e git+https://github.com/NVIDIA-Merlin/NVTabular.git@0f3608c168e39aa4933c856d12dadc4d3f6016a8#egg=nvtabular,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='46473190'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-au539hib
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-au539hib
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=124c7bf2fdf1f081e3c7f999e2d97fcade43298ecdf2d4b3c3d5677fbc480131
  Stored in directory: /tmp/pip-ephem-wheel-cache-lk9d3v5o/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/dataloader.git
  Cloning https://github.com/NVIDIA-Merlin/dataloader.git to /tmp/pip-req-build-u7g6xo43
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/dataloader.git /tmp/pip-req-build-u7g6xo43
  Resolved https://github.com/NVIDIA-Merlin/dataloader.git to commit f1f380cda85c9f4c557971683672397abd0ac040
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.8.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-dataloader==0.0.2+20.gf1f380c) (0.9.0+13.g78f1f0b)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (4.0.0)
Building wheels for collected packages: merlin-dataloader
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2+20.gf1f380c-py3-none-any.whl size=40215 sha256=22a05d7f7825cfc985ca7305385b4b62a54e7a041fc8e3b4a377d08d5a98b81f
  Stored in directory: /tmp/pip-ephem-wheel-cache-inye0ydf/wheels/de/f5/d9/251909f4627d2920fb15548f5ffd6daf1bf24c3c56bb4977b1
Successfully built merlin-dataloader
Installing collected packages: merlin-dataloader
  Attempting uninstall: merlin-dataloader
    Found existing installation: merlin-dataloader 0.0.3
    Uninstalling merlin-dataloader-0.0.3:
      Successfully uninstalled merlin-dataloader-0.0.3
Successfully installed merlin-dataloader-0.0.2+20.gf1f380c
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-y5uezqv7
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-y5uezqv7
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.9.0+13.g78f1f0b)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=fd93e6c5b6deb383560baf7dd8772966f9d71ab9967c6c7913ba221f8822ee7e
  Stored in directory: /tmp/pip-ephem-wheel-cache-y8_2ybwx/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[3] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 1439 items / 1 skipped
tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

....                                                                     [  8%]

tests/unit/test_notebooks.py ..F.                                        [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ................................     [ 12%]

tests/unit/examples/test_01-Getting-started.py .                         [ 12%]

tests/unit/examples/test_02-Advanced-NVTabular-workflow.py .             [ 12%]

tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py F      [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 23%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 25%]

......................................FFFF...........                    [ 29%]

tests/unit/ops/test_categorify.py ...................................... [ 31%]

........................................................................ [ 36%]

.....................................................                    [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 42%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .......................                  [ 47%]

tests/unit/ops/test_hash_bucket.py .........................             [ 49%]

tests/unit/ops/test_join.py ............................................ [ 52%]

........................................................................ [ 57%]

..................................                                       [ 59%]

tests/unit/ops/test_lambda.py ..........                                 [ 60%]

tests/unit/ops/test_normalize.py ....................................... [ 63%]

..                                                                       [ 63%]

tests/unit/ops/test_ops.py ............................................. [ 66%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 70%]

........................................................................ [ 75%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 88%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/ops/test_value_count.py ...                                   [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ................................... [ 92%]

..........................................................               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]
=================================== FAILURES ===================================

____________________________ test_movielens_example ____________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-8/test_movielens_example0')
def test_movielens_example(tmpdir):
    _get_random_movielens_data(tmpdir, 10000, dataset="movie")
    _get_random_movielens_data(tmpdir, 10000, dataset="ratings")
    _get_random_movielens_data(tmpdir, 5000, dataset="ratings", valid=True)

    triton_model_path = os.path.join(tmpdir, "models")
    os.environ["INPUT_DATA_DIR"] = str(tmpdir)
    os.environ["MODEL_PATH"] = triton_model_path

    notebook_path = os.path.join(
        dirname(TEST_PATH),
        "examples/getting-started-movielens/",
        "02-ETL-with-NVTabular.ipynb",
    )
    _run_notebook(tmpdir, notebook_path)

    def _modify_tf_nb(line):
        return line.replace(
            # don't require graphviz/pydot
            "tf.keras.utils.plot_model(model)",
            "# tf.keras.utils.plot_model(model)",
        )

    def _modify_tf_triton(line):
        # models are already preloaded
        line = line.replace("triton_client.load_model", "# triton_client.load_model")
        line = line.replace("triton_client.unload_model", "# triton_client.unload_model")
        return line

    notebooks = []
    try:
        import torch  # noqa

        notebooks.append("03-Training-with-PyTorch.ipynb")
    except Exception:
        pass
    try:
        import nvtabular.inference.triton  # noqa
        import nvtabular.loader.tensorflow  # noqa

        notebooks.append("03-Training-with-TF.ipynb")
        has_tf = True

    except Exception:
        has_tf = False

    for notebook in notebooks:
        notebook_path = os.path.join(
            dirname(TEST_PATH),
            "examples/getting-started-movielens/",
            notebook,
        )
        if notebook == "03-Training-with-TF.ipynb":
            _run_notebook(tmpdir, notebook_path, transform=_modify_tf_nb)
        else:


          _run_notebook(tmpdir, notebook_path)


tests/unit/test_notebooks.py:169:

tests/unit/test_notebooks.py:223: in _run_notebook

subprocess.check_output([sys.executable, script_path])

/usr/lib/python3.8/subprocess.py:415: in check_output

return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

input = None, capture_output = False, timeout = None, check = True

popenargs = (['/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python', '/tmp/pytest-of-jenkins/pytest-8/test_movielens_example0/notebook.py'],)

kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7fbbe377e130>

stdout = b'Total batches: 0\n', stderr = None, retcode = 1
def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:


          raise CalledProcessError(retcode, process.args,


                                     output=stdout, stderr=stderr)

E               subprocess.CalledProcessError: Command '['/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python', '/tmp/pytest-of-jenkins/pytest-8/test_movielens_example0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError

----------------------------- Captured stderr call -----------------------------

Traceback (most recent call last):

File "/tmp/pytest-of-jenkins/pytest-8/test_movielens_example0/notebook.py", line 166, in 

train_loss, y_pred, y = process_epoch(train_loader,

File "/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/framework_utils/torch/utils.py", line 117, in process_epoch

y = torch.cat(y_list)

RuntimeError: torch.cat(): expected a non-empty list of Tensors

_______________________________ test_example_03 ________________________________
def test_example_03():
    with testbook(
        REPO_ROOT / "examples" / "03-Running-on-multiple-GPUs-or-on-CPU.ipynb",
        execute=False,
        timeout=180,
    ) as tb:
        tb.inject(
            """
            import os
            from unittest.mock import patch
            from merlin.datasets.synthetic import generate_data
            mock_train, mock_valid = generate_data(
                input="movielens-1m",
                num_rows=1000,
                set_sizes=(0.8, 0.2)
            )
            input_path = os.environ.get(
                "INPUT_DATA_DIR",
                os.path.expanduser("~/merlin-framework/movielens/")
            )
            from pathlib import Path
            Path(f'{input_path}ml-1m').mkdir(parents=True, exist_ok=True)
            mock_train.compute().to_parquet(f'{input_path}ml-1m/train.parquet')
            mock_train.compute().to_parquet(f'{input_path}ml-1m/valid.parquet')

            p1 = patch(
                "merlin.datasets.entertainment.get_movielens",
                return_value=[mock_train, mock_valid]
            )
            p1.start()

            """
        )


      tb.execute()


tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py:59:

../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/local/lib/python3.8/dist-packages/nest_asyncio.py:89: in run_until_complete

return f.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:280: in __step

result = coro.send(None)

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)

self = <testbook.client.TestbookNotebookClient object at 0x7fbb83c0c6a0>

cell = {'cell_type': 'code', 'execution_count': 8, 'id': '8d204def', 'metadata': {'execution': {'iopub.status.busy': '2022-11...ge)\nexample_workflow.fit_transform(train).to_parquet('train')\nexample_workflow.transform(valid).to_parquet('valid')"}

cell_index = 16

exec_reply = {'buffers': [], 'content': {'ename': 'RuntimeError', 'engine_info': {'engine_id': -1, 'engine_uuid': '19495ff7-84a7-41...e, 'engine': '19495ff7-84a7-41ae-a599-28775807307d', 'started': '2022-11-29T17:54:39.834269Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           categories = ['userId', 'movieId', 'zipcode'] >> nvt.ops.Categorify(freq_threshold=10)

E           age = ['age'] >> nvt.ops.Bucketize([0, 10, 21, 45])

E

E           example_workflow = nvt.Workflow(categories + age)

E           example_workflow.fit_transform(train).to_parquet('train')

E           example_workflow.transform(valid).to_parquet('valid')

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mRuntimeError�[0m                              Traceback (most recent call last)

E           Cell �[0;32mIn [8], line 5�[0m

E           �[1;32m      2�[0m age �[38;5;241m=�[39m [�[38;5;124m'�[39m�[38;5;124mage�[39m�[38;5;124m'�[39m] �[38;5;241m>>�[39m nvt�[38;5;241m.�[39mops�[38;5;241m.�[39mBucketize([�[38;5;241m0�[39m, �[38;5;241m10�[39m, �[38;5;241m21�[39m, �[38;5;241m45�[39m])

E           �[1;32m      4�[0m example_workflow �[38;5;241m=�[39m nvt�[38;5;241m.�[39mWorkflow(categories �[38;5;241m+�[39m age)

E           �[0;32m----> 5�[0m �[43mexample_workflow�[49m�[38;5;241;43m.�[39;49m�[43mfit_transform�[49m�[43m(�[49m�[43mtrain�[49m�[43m)�[49m�[38;5;241m.�[39mto_parquet(�[38;5;124m'�[39m�[38;5;124mtrain�[39m�[38;5;124m'�[39m)

E           �[1;32m      6�[0m example_workflow�[38;5;241m.�[39mtransform(valid)�[38;5;241m.�[39mto_parquet(�[38;5;124m'�[39m�[38;5;124mvalid�[39m�[38;5;124m'�[39m)

E

E           File �[0;32m~/workspace/nvtabular_tests/nvtabular/nvtabular/workflow/workflow.py:234�[0m, in �[0;36mWorkflow.fit_transform�[0;34m(self, dataset)�[0m

E           �[1;32m    214�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mfit_transform�[39m(�[38;5;28mself�[39m, dataset: Dataset) �[38;5;241m-�[39m�[38;5;241m>�[39m Dataset:

E           �[1;32m    215�[0m     �[38;5;124;03m"""Convenience method to both fit the workflow and transform the dataset in a single�[39;00m

E           �[1;32m    216�[0m �[38;5;124;03m    call. Equivalent to calling workflow.fit(dataset) followed by�[39;00m

E           �[1;32m    217�[0m �[38;5;124;03m    workflow.transform(dataset)�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    232�[0m �[38;5;124;03m    transform�[39;00m

E           �[1;32m    233�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m--> 234�[0m     �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mfit�[49m�[43m(�[49m�[43mdataset�[49m�[43m)�[49m

E           �[1;32m    235�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mtransform(dataset)

E

E           File �[0;32m~/workspace/nvtabular_tests/nvtabular/nvtabular/workflow/workflow.py:198�[0m, in �[0;36mWorkflow.fit�[0;34m(self, dataset)�[0m

E           �[1;32m    194�[0m �[38;5;28;01mif�[39;00m �[38;5;129;01mnot�[39;00m current_phase:

E           �[1;32m    195�[0m     �[38;5;66;03m# this shouldn't happen, but lets not infinite loop just in case�[39;00m

E           �[1;32m    196�[0m     �[38;5;28;01mraise�[39;00m �[38;5;167;01mRuntimeError�[39;00m(�[38;5;124m"�[39m�[38;5;124mfailed to find dependency-free StatOperator to fit�[39m�[38;5;124m"�[39m)

E           �[0;32m--> 198�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mexecutor�[49m�[38;5;241;43m.�[39;49m�[43mfit�[49m�[43m(�[49m�[43mddf�[49m�[43m,�[49m�[43m �[49m�[43mcurrent_phase�[49m�[43m)�[49m

E           �[1;32m    200�[0m �[38;5;66;03m# Remove all the operators we processed in this phase, and remove�[39;00m

E           �[1;32m    201�[0m �[38;5;66;03m# from the dependencies of other ops too�[39;00m

E           �[1;32m    202�[0m �[38;5;28;01mfor�[39;00m node �[38;5;129;01min�[39;00m current_phase:

E

E           File �[0;32m~/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334�[0m, in �[0;36mDaskExecutor.fit�[0;34m(self, ddf, nodes)�[0m

E           �[1;32m    332�[0m dask_client �[38;5;241m=�[39m global_dask_client()

E           �[1;32m    333�[0m �[38;5;28;01mif�[39;00m dask_client:

E           �[0;32m--> 334�[0m     results �[38;5;241m=�[39m [r�[38;5;241m.�[39mresult() �[38;5;28;01mfor�[39;00m r �[38;5;129;01min�[39;00m dask_client�[38;5;241m.�[39mcompute(stats)]

E           �[1;32m    335�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    336�[0m     results �[38;5;241m=�[39m dask�[38;5;241m.�[39mcompute(stats, scheduler�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124msynchronous�[39m�[38;5;124m"�[39m)[�[38;5;241m0�[39m]

E

E           File �[0;32m~/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334�[0m, in �[0;36m�[0;34m(.0)�[0m

E           �[1;32m    332�[0m dask_client �[38;5;241m=�[39m global_dask_client()

E           �[1;32m    333�[0m �[38;5;28;01mif�[39;00m dask_client:

E           �[0;32m--> 334�[0m     results �[38;5;241m=�[39m [�[43mr�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m �[38;5;28;01mfor�[39;00m r �[38;5;129;01min�[39;00m dask_client�[38;5;241m.�[39mcompute(stats)]

E           �[1;32m    335�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    336�[0m     results �[38;5;241m=�[39m dask�[38;5;241m.�[39mcompute(stats, scheduler�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124msynchronous�[39m�[38;5;124m"�[39m)[�[38;5;241m0�[39m]

E

E           File �[0;32m~/.local/lib/python3.8/site-packages/distributed/client.py:279�[0m, in �[0;36mFuture.result�[0;34m(self, timeout)�[0m

E           �[1;32m    277�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m==�[39m �[38;5;124m"�[39m�[38;5;124merror�[39m�[38;5;124m"�[39m:

E           �[1;32m    278�[0m     typ, exc, tb �[38;5;241m=�[39m result

E           �[0;32m--> 279�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)

E           �[1;32m    280�[0m �[38;5;28;01melif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m==�[39m �[38;5;124m"�[39m�[38;5;124mcancelled�[39m�[38;5;124m"�[39m:

E           �[1;32m    281�[0m     �[38;5;28;01mraise�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36minner�[0;34m()�[0m

E           �[1;32m     98�[0m �[38;5;129m@wraps�[39m(func)

E           �[1;32m     99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):

E           �[1;32m    100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[0;32m--> 101�[0m     result �[38;5;241m=�[39m func(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)

E           �[1;32m    102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[1;32m    103�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m~/workspace/nvtabular_tests/nvtabular/nvtabular/ops/categorify.py:833�[0m, in �[0;36m_mid_level_groupby�[0;34m()�[0m

E           �[1;32m    829�[0m �[38;5;28;01mif�[39;00m options�[38;5;241m.�[39mon_host �[38;5;129;01mand�[39;00m �[38;5;129;01mnot�[39;00m is_cpu_object(dfs[�[38;5;241m0�[39m]):

E           �[1;32m    830�[0m     �[38;5;66;03m# Construct gpu DataFrame from pyarrow data.�[39;00m

E           �[1;32m    831�[0m     �[38;5;66;03m# on_host=True implies gpu-backed data.�[39;00m

E           �[1;32m    832�[0m     df �[38;5;241m=�[39m pa�[38;5;241m.�[39mconcat_tables(dfs, promote�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E           �[0;32m--> 833�[0m     df �[38;5;241m=�[39m dispatch�[38;5;241m.�[39mfrom_host(df)

E           �[1;32m    834�[0m �[38;5;28;01melse�[39;00m:

E           �[1;32m    835�[0m     df �[38;5;241m=�[39m _concat(dfs, ignore_index�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E

E           File �[0;32m~/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py:605�[0m, in �[0;36mfrom_host�[0;34m()�[0m

E           �[1;32m    603�[0m     �[38;5;28;01mreturn�[39;00m x

E           �[1;32m    604�[0m �[38;5;28;01melse�[39;00m:

E           �[0;32m--> 605�[0m     �[38;5;28;01mreturn�[39;00m cudf�[38;5;241m.�[39mDataFrame�[38;5;241m.�[39mfrom_arrow(x)

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36minner�[0;34m()�[0m

E           �[1;32m     98�[0m �[38;5;129m@wraps�[39m(func)

E           �[1;32m     99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):

E           �[1;32m    100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[0;32m--> 101�[0m     result �[38;5;241m=�[39m func(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)

E           �[1;32m    102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[1;32m    103�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:4615�[0m, in �[0;36mfrom_arrow�[0;34m()�[0m

E           �[1;32m   4610�[0m �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(table, pa�[38;5;241m.�[39mTable) �[38;5;129;01mand�[39;00m �[38;5;28misinstance�[39m(

E           �[1;32m   4611�[0m     table�[38;5;241m.�[39mschema�[38;5;241m.�[39mpandas_metadata, �[38;5;28mdict�[39m

E           �[1;32m   4612�[0m ):

E           �[1;32m   4613�[0m     index_col �[38;5;241m=�[39m table�[38;5;241m.�[39mschema�[38;5;241m.�[39mpandas_metadata[�[38;5;124m"�[39m�[38;5;124mindex_columns�[39m�[38;5;124m"�[39m]

E           �[0;32m-> 4615�[0m out �[38;5;241m=�[39m �[38;5;28msuper�[39m()�[38;5;241m.�[39mfrom_arrow(table)

E           �[1;32m   4617�[0m �[38;5;28;01mif�[39;00m index_col:

E           �[1;32m   4618�[0m     �[38;5;28;01mif�[39;00m �[38;5;28misinstance�[39m(index_col[�[38;5;241m0�[39m], �[38;5;28mdict�[39m):

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101�[0m, in �[0;36minner�[0;34m()�[0m

E           �[1;32m     98�[0m �[38;5;129m@wraps�[39m(func)

E           �[1;32m     99�[0m �[38;5;28;01mdef�[39;00m �[38;5;21minner�[39m(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs):

E           �[1;32m    100�[0m     libnvtx_push_range(�[38;5;28mself�[39m�[38;5;241m.�[39mattributes, �[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[0;32m--> 101�[0m     result �[38;5;241m=�[39m func(�[38;5;241m�[39margs, �[38;5;241m�[39m�[38;5;241m�[39mkwargs)

E           �[1;32m    102�[0m     libnvtx_pop_range(�[38;5;28mself�[39m�[38;5;241m.�[39mdomain�[38;5;241m.�[39mhandle)

E           �[1;32m    103�[0m     �[38;5;28;01mreturn�[39;00m result

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:1418�[0m, in �[0;36mfrom_arrow�[0;34m()�[0m

E           �[1;32m   1401�[0m     cudf_category_frame �[38;5;241m=�[39m {

E           �[1;32m   1402�[0m         name: build_categorical_column(

E           �[1;32m   1403�[0m             cudf_dictionaries_columns[name],

E           �[0;32m   (...)�[0m

E           �[1;32m   1411�[0m         )

E           �[1;32m   1412�[0m     }

E           �[1;32m   1414�[0m �[38;5;66;03m# Handle non-dict arrays�[39;00m

E           �[1;32m   1415�[0m cudf_non_category_frame �[38;5;241m=�[39m {

E           �[1;32m   1416�[0m     name: col

E           �[1;32m   1417�[0m     �[38;5;28;01mfor�[39;00m name, col �[38;5;129;01min�[39;00m �[38;5;28mzip�[39m(

E           �[0;32m-> 1418�[0m         data�[38;5;241m.�[39mcolumn_names, libcudf�[38;5;241m.�[39minterop�[38;5;241m.�[39mfrom_arrow(data)

E           �[1;32m   1419�[0m     )

E           �[1;32m   1420�[0m }

E           �[1;32m   1422�[0m result �[38;5;241m=�[39m {�[38;5;241m�[39m�[38;5;241m�[39mcudf_non_category_frame, �[38;5;241m�[39m�[38;5;241m�[39mcudf_category_frame}

E           �[1;32m   1424�[0m �[38;5;66;03m# There are some special cases that need to be handled�[39;00m

E           �[1;32m   1425�[0m �[38;5;66;03m# based on metadata.�[39;00m

E

E           File �[0;32mcudf/_lib/interop.pyx:150�[0m, in �[0;36mcudf._lib.interop.from_arrow�[0;34m()�[0m

E

E           �[0;31mRuntimeError�[0m: CUDA error encountered at: ../src/bitmask/null_mask.cu:93: 2 cudaErrorMemoryAllocation out of memory

E           RuntimeError: CUDA error encountered at: ../src/bitmask/null_mask.cu:93: 2 cudaErrorMemoryAllocation out of memory
../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-11-29 17:54:38,122 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 17:54:38,156 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 17:54:38,157 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 17:54:38,222 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-11-29 17:54:42,035 - distributed.worker - WARNING - Compute Failed

Key:       ('level_2-3fe3f94678ad7cf6cb6240303ad14b12', 2, 2)

Function:  _mid_level_groupby

args:      ([pyarrow.Table

zipcode: int32

zipcode_size: int32
zipcode: [[396,2069,71,2692,1149,1268,2560,1566,2264,2360,...,1836,1090,1435,1540,679,1872,1665,2794,465,1305]]

zipcode_size: [[2,1,1,1,1,1,1,1,1,1,...,1,1,1,1,1,1,1,2,1,2]]], <merlin.dag.selector.ColumnSelector object at 0x7f7ff02d7b20>, 10, FitOptions(col_groups=[<merlin.dag.selector.ColumnSelector object at 0x7f84547c5d90>, <merlin.dag.selector.ColumnSelector object at 0x7f7fef972580>, <merlin.dag.selector.ColumnSelector object at 0x7f7ff02d7b20>], agg_cols=[], agg_list=['size'], out_path='./', freq_limit=10, tree_width={'userId': 8, 'movieId': 8, 'zipcode': 8}, on_host=True, stat_name='categories', concat_groups=True, name_sep='_', max_size=0, num_buckets=None, start_index=0, cardinality_memory_limit=2133663744))

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/bitmask/null_mask.cu:93: 2 cudaErrorMemoryAllocation out of memory')"
2022-11-29 17:54:42,054 - distributed.worker - WARNING - Compute Failed

Key:       ('level_2-3fe3f94678ad7cf6cb6240303ad14b12', 2, 3)

Function:  _mid_level_groupby

args:      ([pyarrow.Table

zipcode: int32

zipcode_size: int32
zipcode: [[274,1758,1249,2856,1097,1252,2053,1990,1598,3360,...,1869,1620,566,2436,489,115,1507,1744,475,3367]]

zipcode_size: [[2,1,2,1,1,1,1,1,1,1,...,1,1,1,2,1,1,1,2,1,1]]], <merlin.dag.selector.ColumnSelector object at 0x7f845644bf40>, 10, FitOptions(col_groups=[<merlin.dag.selector.ColumnSelector object at 0x7f84547c5b50>, <merlin.dag.selector.ColumnSelector object at 0x7f7ff01eba00>, <merlin.dag.selector.ColumnSelector object at 0x7f845644bf40>], agg_cols=[], agg_list=['size'], out_path='./', freq_limit=10, tree_width={'userId': 8, 'movieId': 8, 'zipcode': 8}, on_host=True, stat_name='categories', concat_groups=True, name_sep='_', max_size=0, num_buckets=None, start_index=0, cardinality_memory_limit=2133663744))

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/bitmask/null_mask.cu:93: 2 cudaErrorMemoryAllocation out of memory')"
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

__________________ test_gpu_dl_break[None-parquet-1000-0.001] __________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-8/test_gpu_dl_break_None_parquet0')

df =       name-cat name-string    id  label         x         y

0        Zelda     Michael  1005   1079  0.349774 -0.43511...n   951   1005  0.593635 -0.878707

4320    Ursula       Sarah   994   1060 -0.579977  0.275699
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fbb587a6220>

batch_size = 1000, part_mem_fraction = 0.001, engine = 'parquet', device = None
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

__________________ test_gpu_dl_break[None-parquet-1000-0.06] ___________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-8/test_gpu_dl_break_None_parquet1')

df =       name-cat name-string    id  label         x         y

0        Zelda     Michael  1005   1079  0.349774 -0.43511...n   951   1005  0.593635 -0.878707

4320    Ursula       Sarah   994   1060 -0.579977  0.275699
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fbb9b15f7c0>

batch_size = 1000, part_mem_fraction = 0.06, engine = 'parquet', device = None
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

___________________ test_gpu_dl_break[0-parquet-1000-0.001] ____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-8/test_gpu_dl_break_0_parquet_100')

df =       name-cat name-string    id  label         x         y

0        Zelda     Michael  1005   1079  0.349774 -0.43511...n   951   1005  0.593635 -0.878707

4320    Ursula       Sarah   994   1060 -0.579977  0.275699
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fbb587bcd90>

batch_size = 1000, part_mem_fraction = 0.001, engine = 'parquet', device = 0
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

____________________ test_gpu_dl_break[0-parquet-1000-0.06] ____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-8/test_gpu_dl_break_0_parquet_101')

df =       name-cat name-string    id  label         x         y

0        Zelda     Michael  1005   1079  0.349774 -0.43511...n   951   1005  0.593635 -0.878707

4320    Ursula       Sarah   994   1060 -0.579977  0.275699
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fbb5863b4c0>

batch_size = 1000, part_mem_fraction = 0.06, engine = 'parquet', device = 0
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
tests/unit/test_dask_nvt.py: 6 warnings

tests/unit/workflow/test_workflow.py: 78 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/dask/base.py:1282: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.

warnings.warn(
tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py:431: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(
tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                Stmts   Miss  Cover
merlin/transforms/init.py           1      1     0%

merlin/transforms/ops/init.py       1      1     0%
TOTAL                                   2      2     0%
=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:14: could not import 'moto': No module named 'moto'

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

===== 6 failed, 1432 passed, 2 skipped, 147 warnings in 1327.85s (0:22:07) =====

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Data file '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.coverage.10.20.17.231.27700.385647' doesn't seem to be a coverage data file: cannot unpack non-iterable NoneType object

data._warn(str(exc))

ERROR: InvocationError for command /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[nvtabular_tests] $ /bin/bash /tmp/jenkins3268966066333387406.sh

mikemckiernan · 2022-11-30T02:34:11Z

rerun tests

nvidia-merlin-bot · 2022-11-30T02:56:14Z

Click to view CI Results

GitHub pull request #1719 of commit 0f3608c168e39aa4933c856d12dadc4d3f6016a8, no merge conflicts.
GitHub pull request #1719 of commit 0f3608c168e39aa4933c856d12dadc4d3f6016a8, no merge conflicts.
Running as SYSTEM
Setting status of 0f3608c168e39aa4933c856d12dadc4d3f6016a8 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/nvtabular_tests/4909/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
 > git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1719/*:refs/remotes/origin/pr/1719/* # timeout=10
 > git rev-parse 0f3608c168e39aa4933c856d12dadc4d3f6016a8^{commit} # timeout=10
Checking out Revision 0f3608c168e39aa4933c856d12dadc4d3f6016a8 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0f3608c168e39aa4933c856d12dadc4d3f6016a8 # timeout=10
Commit message: "Describe how to check for broken links"
 > git rev-list --no-walk 0f3608c168e39aa4933c856d12dadc4d3f6016a8 # timeout=10
[nvtabular_tests] $ /bin/bash /tmp/jenkins11195154817249845853.sh
GLOB sdist-make: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/setup.py
test-gpu create: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/.tmp/package/1/nvtabular-1.6.0+12.g0f3608c1.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.3,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,-e git+https://github.com/NVIDIA-Merlin/NVTabular.git@0f3608c168e39aa4933c856d12dadc4d3f6016a8#egg=nvtabular,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3532060263'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-49blyphb
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-49blyphb
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=0974f4a1ab517fd71ba97697cc79c0cc3b87e1ace20a62ddccae036ae0643c03
  Stored in directory: /tmp/pip-ephem-wheel-cache-w_6mwmbk/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/dataloader.git
  Cloning https://github.com/NVIDIA-Merlin/dataloader.git to /tmp/pip-req-build-bx83axdb
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/dataloader.git /tmp/pip-req-build-bx83axdb
  Resolved https://github.com/NVIDIA-Merlin/dataloader.git to commit f1f380cda85c9f4c557971683672397abd0ac040
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.8.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-dataloader==0.0.2+20.gf1f380c) (0.9.0+13.g78f1f0b)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (5.4.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+20.gf1f380c) (6.0.1)
Building wheels for collected packages: merlin-dataloader
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2+20.gf1f380c-py3-none-any.whl size=40215 sha256=1da68a8c404ca9f994a4081336eb492a3b4f80acff55e72e4b687af778794be3
  Stored in directory: /tmp/pip-ephem-wheel-cache-un8iee3f/wheels/de/f5/d9/251909f4627d2920fb15548f5ffd6daf1bf24c3c56bb4977b1
Successfully built merlin-dataloader
Installing collected packages: merlin-dataloader
  Attempting uninstall: merlin-dataloader
    Found existing installation: merlin-dataloader 0.0.3
    Uninstalling merlin-dataloader-0.0.3:
      Successfully uninstalled merlin-dataloader-0.0.3
Successfully installed merlin-dataloader-0.0.2+20.gf1f380c
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-c92bvfmh
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-c92bvfmh
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 3124b03033723de762d90d47085bac3191f37ff5
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.9.0+59.g3124b030) (0.9.0+13.g78f1f0b)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.4.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.7.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+59.g3124b030) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+59.g3124b030-py3-none-any.whl size=366427 sha256=6d3aa363995458d91eb34e7e750c8aee8cf3468cf65f76390f74713931e7c260
  Stored in directory: /tmp/pip-ephem-wheel-cache-o4mn6uf4/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+59.g3124b030
test-gpu run-test: commands[3] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 1439 items / 1 skipped
tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

....                                                                     [  8%]

tests/unit/test_notebooks.py ..F.                                        [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ..............................FF     [ 12%]

tests/unit/examples/test_01-Getting-started.py .                         [ 12%]

tests/unit/examples/test_02-Advanced-NVTabular-workflow.py .             [ 12%]

tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py .      [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 23%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 25%]

......................................FFFF...........                    [ 29%]

tests/unit/ops/test_categorify.py ...................................... [ 31%]

........................................................................ [ 36%]

.....................................................                    [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 42%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .......................                  [ 47%]

tests/unit/ops/test_hash_bucket.py .........................             [ 49%]

tests/unit/ops/test_join.py ............................................ [ 52%]

........................................................................ [ 57%]

..................................                                       [ 59%]

tests/unit/ops/test_lambda.py ..........                                 [ 60%]

tests/unit/ops/test_normalize.py ....................................... [ 63%]

..                                                                       [ 63%]

tests/unit/ops/test_ops.py ............................................. [ 66%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 70%]

........................................................................ [ 75%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 88%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/ops/test_value_count.py ...                                   [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ................................... [ 92%]

..........................................................               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]
=================================== FAILURES ===================================

____________________________ test_movielens_example ____________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_movielens_example0')
def test_movielens_example(tmpdir):
    _get_random_movielens_data(tmpdir, 10000, dataset="movie")
    _get_random_movielens_data(tmpdir, 10000, dataset="ratings")
    _get_random_movielens_data(tmpdir, 5000, dataset="ratings", valid=True)

    triton_model_path = os.path.join(tmpdir, "models")
    os.environ["INPUT_DATA_DIR"] = str(tmpdir)
    os.environ["MODEL_PATH"] = triton_model_path

    notebook_path = os.path.join(
        dirname(TEST_PATH),
        "examples/getting-started-movielens/",
        "02-ETL-with-NVTabular.ipynb",
    )
    _run_notebook(tmpdir, notebook_path)

    def _modify_tf_nb(line):
        return line.replace(
            # don't require graphviz/pydot
            "tf.keras.utils.plot_model(model)",
            "# tf.keras.utils.plot_model(model)",
        )

    def _modify_tf_triton(line):
        # models are already preloaded
        line = line.replace("triton_client.load_model", "# triton_client.load_model")
        line = line.replace("triton_client.unload_model", "# triton_client.unload_model")
        return line

    notebooks = []
    try:
        import torch  # noqa

        notebooks.append("03-Training-with-PyTorch.ipynb")
    except Exception:
        pass
    try:
        import nvtabular.inference.triton  # noqa
        import nvtabular.loader.tensorflow  # noqa

        notebooks.append("03-Training-with-TF.ipynb")
        has_tf = True

    except Exception:
        has_tf = False

    for notebook in notebooks:
        notebook_path = os.path.join(
            dirname(TEST_PATH),
            "examples/getting-started-movielens/",
            notebook,
        )
        if notebook == "03-Training-with-TF.ipynb":
            _run_notebook(tmpdir, notebook_path, transform=_modify_tf_nb)
        else:


          _run_notebook(tmpdir, notebook_path)


tests/unit/test_notebooks.py:169:

tests/unit/test_notebooks.py:223: in _run_notebook

subprocess.check_output([sys.executable, script_path])

/usr/lib/python3.8/subprocess.py:415: in check_output

return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

input = None, capture_output = False, timeout = None, check = True

popenargs = (['/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python', '/tmp/pytest-of-jenkins/pytest-17/test_movielens_example0/notebook.py'],)

kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7fcdbc31bac0>

stdout = b'Total batches: 0\n', stderr = None, retcode = 1
def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:


          raise CalledProcessError(retcode, process.args,


                                     output=stdout, stderr=stderr)

E               subprocess.CalledProcessError: Command '['/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python', '/tmp/pytest-of-jenkins/pytest-17/test_movielens_example0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError

----------------------------- Captured stderr call -----------------------------

Traceback (most recent call last):

File "/tmp/pytest-of-jenkins/pytest-17/test_movielens_example0/notebook.py", line 166, in 

train_loss, y_pred, y = process_epoch(train_loader,

File "/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/nvtabular/framework_utils/torch/utils.py", line 117, in process_epoch

y = torch.cat(y_list)

RuntimeError: torch.cat(): expected a non-empty list of Tensors

_________________________ test_groupby_model[pytorch] __________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_groupby_model_pytorch_0')

output_model = 'pytorch'
@pytest.mark.skipif(TRITON_SERVER_PATH is None, reason="Requires tritonserver on the path")
@pytest.mark.parametrize("output_model", ["tensorflow", "pytorch"])
def test_groupby_model(tmpdir, output_model):
    size = 20
    df = make_df(
        {
            "id": np.random.choice([0, 1], size=size),
            "ts": np.linspace(0.0, 10.0, num=size),
            "x": np.arange(size),
            "y": np.linspace(0.0, 10.0, num=size),
        }
    )

    groupby_features = ColumnSelector(["id", "ts", "x", "y"]) >> ops.Groupby(
        groupby_cols=["id"],
        sort_cols=["ts"],
        aggs={
            "x": ["sum"],
            "y": ["first"],
        },
        name_sep="-",
    )
    workflow = nvt.Workflow(groupby_features)


  _verify_workflow_on_tritonserver(


        tmpdir, workflow, df, "groupby", output_model, cats=["id", "y-first"], conts=["x-sum"]
    )

tests/unit/test_triton_inference.py:379:

tests/unit/test_triton_inference.py:111: in _verify_workflow_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

modelpath = local('/tmp/pytest-of-jenkins/pytest-17/test_groupby_model_pytorch_0')
@contextlib.contextmanager
def run_triton_server(modelpath):
    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        modelpath,
        "--backend-config=tensorflow,version=2",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient("localhost:8001") as client:
                # wait until server is ready
                for _ in range(60):
                    if process.poll() is not None:
                        retcode = process.returncode
                        raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")

                    try:
                        ready = client.is_server_ready()
                    except tritonclient.utils.InferenceServerException:
                        ready = False

                    if ready:
                        yield client
                        return

                    time.sleep(1)


              raise RuntimeError("Timed out waiting for tritonserver to become ready")


E                   RuntimeError: Timed out waiting for tritonserver to become ready
tests/unit/test_triton_inference.py:62: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1130 02:41:51.047647 18703 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7faa24000000' with size 268435456

I1130 02:41:51.048436 18703 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1130 02:41:51.051664 18703 model_lifecycle.cc:459] loading: groupby:1

______________________ test_seq_etl_tf_model[tensorflow] _______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_seq_etl_tf_model_tensorfl0')

output_model = 'tensorflow'
@pytest.mark.skipif(TRITON_SERVER_PATH is None, reason="Requires tritonserver on the path")
@pytest.mark.parametrize("output_model", ["tensorflow"])
def test_seq_etl_tf_model(tmpdir, output_model):
    size = 100
    max_length = 10
    df = make_df(
        {
            "id": np.random.choice([0, 1], size=size),
            "item_id": np.random.randint(1, 10, size),
            "ts": np.linspace(0.0, 10.0, num=size).astype(np.float32),
            "y": np.linspace(0.0, 10.0, num=size).astype(np.float32),
        }
    )

    groupby_features = ColumnSelector(["id", "item_id", "ts", "y"]) >> ops.Groupby(
        groupby_cols=["id"],
        sort_cols=["ts"],
        aggs={
            "item_id": ["list"],
            "y": ["list"],
        },
        name_sep="-",
    )
    feats_list = groupby_features["item_id-list", "y-list"]
    feats_trim = feats_list >> ops.ListSlice(0, max_length, pad=True)
    selected_features = groupby_features["id"] + feats_trim

    workflow = nvt.Workflow(selected_features)

    sparse_max = {"item_id-list": max_length, "y-list": max_length}


  _verify_workflow_on_tritonserver(


        tmpdir,
        workflow,
        df,
        "groupby",
        output_model,
        sparse_max,
        cats=["id", "item_id-list"],
        conts=["y-list"],
    )

tests/unit/test_triton_inference.py:415:

tests/unit/test_triton_inference.py:111: in _verify_workflow_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

modelpath = local('/tmp/pytest-of-jenkins/pytest-17/test_seq_etl_tf_model_tensorfl0')
@contextlib.contextmanager
def run_triton_server(modelpath):
    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        modelpath,
        "--backend-config=tensorflow,version=2",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient("localhost:8001") as client:
                # wait until server is ready
                for _ in range(60):
                    if process.poll() is not None:
                        retcode = process.returncode
                        raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")

                    try:
                        ready = client.is_server_ready()
                    except tritonclient.utils.InferenceServerException:
                        ready = False

                    if ready:
                        yield client
                        return

                    time.sleep(1)


              raise RuntimeError("Timed out waiting for tritonserver to become ready")


E                   RuntimeError: Timed out waiting for tritonserver to become ready
tests/unit/test_triton_inference.py:62: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1130 02:42:51.473010 18784 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f9f56000000' with size 268435456

I1130 02:42:51.473757 18784 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1130 02:42:51.476857 18784 model_lifecycle.cc:459] loading: groupby:1

__________________ test_gpu_dl_break[None-parquet-1000-0.001] __________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_gpu_dl_break_None_parquet0')

df =       name-cat name-string    id  label         x         y

0       Hannah         Ray  1026   1080 -0.261469  0.99823...k  1056   1034 -0.088151 -0.297846

4320  Patricia      Ursula  1003    976 -0.329359 -0.266414
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fcd7015bf40>

batch_size = 1000, part_mem_fraction = 0.001, engine = 'parquet', device = None
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

__________________ test_gpu_dl_break[None-parquet-1000-0.06] ___________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_gpu_dl_break_None_parquet1')

df =       name-cat name-string    id  label         x         y

0       Hannah         Ray  1026   1080 -0.261469  0.99823...k  1056   1034 -0.088151 -0.297846

4320  Patricia      Ursula  1003    976 -0.329359 -0.266414
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fcd8c6ec370>

batch_size = 1000, part_mem_fraction = 0.06, engine = 'parquet', device = None
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

___________________ test_gpu_dl_break[0-parquet-1000-0.001] ____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_gpu_dl_break_0_parquet_100')

df =       name-cat name-string    id  label         x         y

0       Hannah         Ray  1026   1080 -0.261469  0.99823...k  1056   1034 -0.088151 -0.297846

4320  Patricia      Ursula  1003    976 -0.329359 -0.266414
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fcd70193ac0>

batch_size = 1000, part_mem_fraction = 0.001, engine = 'parquet', device = 0
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

____________________ test_gpu_dl_break[0-parquet-1000-0.06] ____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-17/test_gpu_dl_break_0_parquet_101')

df =       name-cat name-string    id  label         x         y

0       Hannah         Ray  1026   1080 -0.261469  0.99823...k  1056   1034 -0.088151 -0.297846

4320  Patricia      Ursula  1003    976 -0.329359 -0.266414
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7fcd8c71a910>

batch_size = 1000, part_mem_fraction = 0.06, engine = 'parquet', device = 0
@pytest.mark.parametrize("part_mem_fraction", [0.001, 0.06])
@pytest.mark.parametrize("batch_size", [1000])
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("device", [None, 0])
def test_gpu_dl_break(tmpdir, df, dataset, batch_size, part_mem_fraction, engine, device):
    cat_names = ["name-cat", "name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    conts = cont_names >> ops.FillMedian() >> ops.Normalize()
    cats = cat_names >> ops.Categorify()

    processor = nvt.Workflow(conts + cats + label_name)

    output_train = os.path.join(tmpdir, "train/")
    os.mkdir(output_train)

    processor.fit_transform(dataset).to_parquet(
        shuffle=nvt.io.Shuffle.PER_PARTITION,
        output_path=output_train,
        out_files_per_proc=2,
    )

    tar_paths = [
        os.path.join(output_train, x) for x in os.listdir(output_train) if x.endswith("parquet")
    ]

    nvt_data = nvt.Dataset(tar_paths[0], engine="parquet", part_mem_fraction=part_mem_fraction)
    data_itr = torch_dataloader.TorchAsyncItr(
        nvt_data,
        batch_size=batch_size,
        cats=cat_names,
        conts=cont_names,
        labels=["label"],
        device=device,
    )
    len_dl = len(data_itr) - 1

    first_chunk = 0
    idx = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk = len(chunk[0])
        last_chk = len(chunk[0])
        print(last_chk)
        if idx == 1:
            break
        del chunk

    assert idx < len_dl

    first_chunk_2 = 0
    for idx, chunk in enumerate(data_itr):
        if idx == 0:
            first_chunk_2 = len(chunk[0])
        del chunk


  assert idx == len_dl


E       assert 0 == 2
tests/unit/loader/test_torch_dataloader.py:301: AssertionError

----------------------------- Captured stdout call -----------------------------

5

5

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
tests/unit/test_dask_nvt.py: 6 warnings

tests/unit/workflow/test_workflow.py: 78 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/dask/base.py:1282: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.

warnings.warn(
tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py:431: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(
tests/unit/test_notebooks.py::test_optimize_criteo

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 33931 instead

warnings.warn(
tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                Stmts   Miss  Cover
merlin/transforms/init.py           1      1     0%

merlin/transforms/ops/init.py       1      1     0%
TOTAL                                   2      2     0%
=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:14: could not import 'moto': No module named 'moto'

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

===== 7 failed, 1431 passed, 2 skipped, 148 warnings in 1216.46s (0:20:16) =====

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[nvtabular_tests] $ /bin/bash /tmp/jenkins4680762567516898709.sh

github-actions · 2022-12-01T16:13:29Z

Documentation preview

https://nvidia-merlin.github.io/NVTabular/review/pr-1719

nvidia-merlin-bot · 2022-12-01T16:39:35Z

Click to view CI Results

GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
Running as SYSTEM
Setting status of 05c2b8c3737512cff6262a6a952ebfb2ec99f860 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/nvtabular_tests/4941/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on the built-in node in workspace /var/jenkins_home/jobs/nvtabular_tests/workspace
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
 > git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1719/*:refs/remotes/origin/pr/1719/* # timeout=10
 > git rev-parse 05c2b8c3737512cff6262a6a952ebfb2ec99f860^{commit} # timeout=10
Checking out Revision 05c2b8c3737512cff6262a6a952ebfb2ec99f860 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 05c2b8c3737512cff6262a6a952ebfb2ec99f860 # timeout=10
Commit message: "Merge branch 'main' into dev-doc-link-testing"
 > git rev-list --no-walk 5133753e943a91286fa0480a610d839b298e0aee # timeout=10
First time build. Skipping changelog.
[workspace] $ /bin/bash /tmp/jenkins9977885169448087682.sh
GLOB sdist-make: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/setup.py
test-gpu create: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/.tmp/package/1/nvtabular-1.6.0+14.g05c2b8c3.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.3,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,-e git+https://github.com/NVIDIA-Merlin/NVTabular.git@05c2b8c3737512cff6262a6a952ebfb2ec99f860#egg=nvtabular,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3134169996'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-x99p92z7
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-x99p92z7
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=7729475bb74837a587e4465994d3c11ab3eacc0de0ead548ddc80d975ad632a6
  Stored in directory: /tmp/pip-ephem-wheel-cache-2lfucmmh/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/dataloader.git
  Cloning https://github.com/NVIDIA-Merlin/dataloader.git to /tmp/pip-req-build-q1og7wr5
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/dataloader.git /tmp/pip-req-build-q1og7wr5
  Resolved https://github.com/NVIDIA-Merlin/dataloader.git to commit 2783231f3aa39dd6acb0e5f6431cc73faacde7fe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.8.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-dataloader==0.0.2+21.g2783231) (0.9.0+14.g4f73ff5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+21.g2783231) (4.0.0)
Building wheels for collected packages: merlin-dataloader
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2+21.g2783231-py3-none-any.whl size=40305 sha256=a74e557441ae8bcdc78d66e9e2b036025a424ddc62d4dee9618f45d0a066f1e7
  Stored in directory: /tmp/pip-ephem-wheel-cache-rvi65inh/wheels/de/f5/d9/251909f4627d2920fb15548f5ffd6daf1bf24c3c56bb4977b1
Successfully built merlin-dataloader
Installing collected packages: merlin-dataloader
  Attempting uninstall: merlin-dataloader
    Found existing installation: merlin-dataloader 0.0.3
    Uninstalling merlin-dataloader-0.0.3:
      Successfully uninstalled merlin-dataloader-0.0.3
Successfully installed merlin-dataloader-0.0.2+21.g2783231
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-lavey3ca
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-lavey3ca
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit 5e4d8a106498847b5755f94a8e467c46cbf12c60
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.9.0+60.g5e4d8a10) (0.9.0+14.g4f73ff5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+60.g5e4d8a10) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+60.g5e4d8a10-py3-none-any.whl size=366424 sha256=67a5ad25f6ced3e6907535a36312391a84542f854c2af59900a4dd8bff8f6676
  Stored in directory: /tmp/pip-ephem-wheel-cache-ollumdq0/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+60.g5e4d8a10
test-gpu run-test: commands[3] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 1439 items / 1 skipped
tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

....                                                                     [  8%]

tests/unit/test_notebooks.py ..Build timed out (after 60 minutes). Marking the build as failed.

FERROR: Got SIGTERM, handling it as a KeyboardInterrupt

ERROR: got KeyboardInterrupt signal

Build was aborted

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins1122160770830057800.sh

mikemckiernan · 2022-12-01T17:24:38Z

rerun tests

nvidia-merlin-bot · 2022-12-01T17:44:19Z

Click to view CI Results

GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
Running as SYSTEM
Setting status of 05c2b8c3737512cff6262a6a952ebfb2ec99f860 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/nvtabular_tests/4942/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on the built-in node in workspace /var/jenkins_home/jobs/nvtabular_tests/workspace
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1719/*:refs/remotes/origin/pr/1719/* # timeout=10
 > git rev-parse 05c2b8c3737512cff6262a6a952ebfb2ec99f860^{commit} # timeout=10
Checking out Revision 05c2b8c3737512cff6262a6a952ebfb2ec99f860 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 05c2b8c3737512cff6262a6a952ebfb2ec99f860 # timeout=10
Commit message: "Merge branch 'main' into dev-doc-link-testing"
 > git rev-list --no-walk 05c2b8c3737512cff6262a6a952ebfb2ec99f860 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins3907002479392737768.sh
GLOB sdist-make: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/setup.py
test-gpu recreate: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/.tmp/package/1/nvtabular-1.6.0+14.g05c2b8c3.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.3,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,-e git+https://github.com/NVIDIA-Merlin/NVTabular.git@05c2b8c3737512cff6262a6a952ebfb2ec99f860#egg=nvtabular,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1758288963'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-92nytdb8
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-92nytdb8
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=ee174483cd4b9911c8ca058c657c620b622ffa1dac0b286247e0413b48fe1424
  Stored in directory: /tmp/pip-ephem-wheel-cache-hu5dz58r/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/dataloader.git
  Cloning https://github.com/NVIDIA-Merlin/dataloader.git to /tmp/pip-req-build-2qi_hy67
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/dataloader.git /tmp/pip-req-build-2qi_hy67
  Resolved https://github.com/NVIDIA-Merlin/dataloader.git to commit 498543d54629216dc09a5854dca4dbeeabcab356
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.8.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-dataloader==0.0.2+22.g498543d) (0.9.0+14.g4f73ff5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.12.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.0.1)
Building wheels for collected packages: merlin-dataloader
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2+22.g498543d-py3-none-any.whl size=40350 sha256=9248700922a4792a9e30ed4f0fe120cbb1e61ad9955359b65581402c06678d67
  Stored in directory: /tmp/pip-ephem-wheel-cache-m1zn86xr/wheels/de/f5/d9/251909f4627d2920fb15548f5ffd6daf1bf24c3c56bb4977b1
Successfully built merlin-dataloader
Installing collected packages: merlin-dataloader
  Attempting uninstall: merlin-dataloader
    Found existing installation: merlin-dataloader 0.0.3
    Uninstalling merlin-dataloader-0.0.3:
      Successfully uninstalled merlin-dataloader-0.0.3
Successfully installed merlin-dataloader-0.0.2+22.g498543d
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-hygjb2qz
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-hygjb2qz
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit e08a72c9c59416a9000e62d25548eb08367fc3fa
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.9.0+61.ge08a72c9) (0.9.0+14.g4f73ff5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.12.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.4)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+61.ge08a72c9-py3-none-any.whl size=367208 sha256=a8a5d801a0754bcbd14cbd9c184414615ef42f369af9243fe5128aa034efd808
  Stored in directory: /tmp/pip-ephem-wheel-cache-tvbqtxcx/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+61.ge08a72c9
test-gpu run-test: commands[3] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 1439 items / 1 skipped
tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

....                                                                     [  8%]

tests/unit/test_notebooks.py ....                                        [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ................................     [ 12%]

tests/unit/examples/test_01-Getting-started.py .                         [ 12%]

tests/unit/examples/test_02-Advanced-NVTabular-workflow.py .             [ 12%]

tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py .      [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 23%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 25%]

.....................................................                    [ 29%]

tests/unit/ops/test_categorify.py ...................................... [ 31%]

........................................................................ [ 36%]

.....................................................                    [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 42%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .......................                  [ 47%]

tests/unit/ops/test_hash_bucket.py .........................             [ 49%]

tests/unit/ops/test_join.py ............................................ [ 52%]

........................................................................ [ 57%]

..................................                                       [ 59%]

tests/unit/ops/test_lambda.py ..........                                 [ 60%]

tests/unit/ops/test_normalize.py ....................................... [ 63%]

..                                                                       [ 63%]

tests/unit/ops/test_ops.py ............................................. [ 66%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 70%]

........................................................................ [ 75%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 88%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/ops/test_value_count.py ...                                   [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ...FFEF.FFFEEFF..EE....EE..E.FFFEEF [ 92%]

FFFEE..F.EE..FFEEFFF.EE.FFFEE.F.........................F.               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]
==================================== ERRORS ====================================

_________ ERROR at setup of test_gpu_workflow_api[True-True-csv-0.01] __________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[True-True-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

---------------------------- Captured stderr setup -----------------------------

2022-12-01 17:41:39,439 - distributed.worker - WARNING - Compute Failed

Key:       ('read-csv-1de58f54ef56bb35810958a9255b04ab', 0)

Function:  _read_csv

args:      ('/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', array([dtype('O'), dtype('int64'), dtype('int64'), dtype('O'),

dtype('float64'), dtype('float64'), dtype('float64')], dtype=object))

kwargs:    {'storage_options': {}, 'byte_range': (0, 170693099)}

Exception: "MemoryError('std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory')"
_________ ERROR at setup of test_gpu_workflow_api[True-False-csv-0.01] _________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[True-False-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_________ ERROR at setup of test_gpu_workflow_api[True-False-csv-0.1] __________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[True-False-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_________ ERROR at setup of test_gpu_workflow_api[False-True-csv-0.01] _________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[False-True-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_________ ERROR at setup of test_gpu_workflow_api[False-True-csv-0.1] __________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[False-True-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

________ ERROR at setup of test_gpu_workflow_api[False-False-csv-0.01] _________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[False-False-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_________ ERROR at setup of test_gpu_workflow_api[False-False-csv-0.1] _________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_api[False-False-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_____________ ERROR at setup of test_gpu_dataset_iterator_csv[csv] _____________
request = <SubRequest 'dataset' for <Function test_gpu_dataset_iterator_csv[csv]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

______________ ERROR at setup of test_gpu_workflow[True-csv-0.01] ______________
request = <SubRequest 'dataset' for <Function test_gpu_workflow[True-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

______________ ERROR at setup of test_gpu_workflow[True-csv-0.1] _______________
request = <SubRequest 'dataset' for <Function test_gpu_workflow[True-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_____________ ERROR at setup of test_gpu_workflow[False-csv-0.01] ______________
request = <SubRequest 'dataset' for <Function test_gpu_workflow[False-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

______________ ERROR at setup of test_gpu_workflow[False-csv-0.1] ______________
request = <SubRequest 'dataset' for <Function test_gpu_workflow[False-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

________ ERROR at setup of test_gpu_workflow_config[True-True-csv-0.01] ________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[True-True-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

________ ERROR at setup of test_gpu_workflow_config[True-True-csv-0.1] _________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[True-True-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_______ ERROR at setup of test_gpu_workflow_config[True-False-csv-0.01] ________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[True-False-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

________ ERROR at setup of test_gpu_workflow_config[True-False-csv-0.1] ________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[True-False-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_______ ERROR at setup of test_gpu_workflow_config[False-True-csv-0.01] ________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[False-True-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

________ ERROR at setup of test_gpu_workflow_config[False-True-csv-0.1] ________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[False-True-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_______ ERROR at setup of test_gpu_workflow_config[False-False-csv-0.01] _______
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[False-False-csv-0.01]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

_______ ERROR at setup of test_gpu_workflow_config[False-False-csv-0.1] ________
request = <SubRequest 'dataset' for <Function test_gpu_workflow_config[False-False-csv-0.1]>>

paths = ['/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', '/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-1.csv']

engine = 'csv'
@pytest.fixture(scope="function")
def dataset(request, paths, engine):
    try:
        gpu_memory_frac = request.getfixturevalue("gpu_memory_frac")
    except Exception:  # pylint: disable=broad-except
        gpu_memory_frac = 0.01

    try:
        cpu = request.getfixturevalue("cpu")
    except Exception:  # pylint: disable=broad-except
        cpu = False

    kwargs = {}
    if engine == "csv-no-header":
        kwargs["names"] = allcols_csv


  return nvtabular.Dataset(paths, part_mem_fraction=gpu_memory_frac, cpu=cpu, **kwargs)


tests/conftest.py:224:

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:348: in init

self.infer_schema()

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1131: in infer_schema

dtypes = self.sample_dtypes(n=n, annotate_lists=True)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset.py:1151: in sample_dtypes

_real_meta = self.engine.sample_data(n=n)

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/dataset_engine.py:71: in sample_data

_head = _ddf.partitions[partition_index].head(n)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1196: in head

return self._head(n=n, npartitions=npartitions, compute=compute, safe=safe)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:1230: in _head

result = result.compute()

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

=================================== FAILURES ===================================

________________ test_gpu_workflow_api[True-True-parquet-0.01] _________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Tru0')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2cdc1596d0>

gpu_memory_frac = 0.01, engine = 'parquet', dump = True, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/dask/optimization.py:990: in call

return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))

../../../.local/lib/python3.8/site-packages/dask/core.py:149: in get

result = _execute_task(task, cache)

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in _execute_task

return func((_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in 

return func((_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/core.py:113: in _execute_task

return [_execute_task(a, cache) for a in arg]

../../../.local/lib/python3.8/site-packages/dask/core.py:113: in 

return [_execute_task(a, cache) for a in arg]

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in _execute_task

return func(*(_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py:82: in call

return read_parquet_part(

../../../.local/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py:490: in read_parquet_part

dfs = [

../../../.local/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py:491: in 

func(fs, rg, columns.copy(), index, **toolz.merge(kwargs, kw))

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py:128: in read_partition

return CudfEngine.read_partition(fs, pieces, *args, **kwargs)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/parquet.py:241: in read_partition

cls._read_paths(

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/parquet.py:92: in _read_paths

df = cudf.read_parquet(

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:470: in read_parquet

return _parquet_to_frame(

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:499: in _parquet_to_frame

return _read_parquet(

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/parquet.py:576: in _read_parquet

return libparquet.read_parquet(

cudf/_lib/parquet.pyx:113: in cudf._lib.parquet.read_parquet

???


???

E   RuntimeError: CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory

cudf/_lib/parquet.pyx:173: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:36,269 - distributed.worker - WARNING - Compute Failed

Key:       ('transform-de706c0319a4bcdf181f137e3e5e61c4', 1)

Function:  subgraph_callable-03e9d2b4-edad-4fa5-8c15-63c4080b

args:      ({'piece': ('/tmp/pytest-of-jenkins/pytest-6/parquet0/dataset-1.parquet', [0], [])})

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"
2022-12-01 17:41:36,281 - distributed.worker - WARNING - Compute Failed

Key:       ('transform-efcba79e4267ced2f9402c48794fe075', 0)

Function:  subgraph_callable-149ceb44-07c8-4434-890c-49e4db40

args:      ({'piece': ('/tmp/pytest-of-jenkins/pytest-6/parquet0/dataset-0.parquet', [0], [])})

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"
_________________ test_gpu_workflow_api[True-True-parquet-0.1] _________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Tru1')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2985885700>

gpu_memory_frac = 0.1, engine = 'parquet', dump = True, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/dask/optimization.py:990: in call

return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))

../../../.local/lib/python3.8/site-packages/dask/core.py:149: in get

result = _execute_task(task, cache)

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in _execute_task

return func((_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in 

return func((_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/core.py:113: in _execute_task

return [_execute_task(a, cache) for a in arg]

../../../.local/lib/python3.8/site-packages/dask/core.py:113: in 

return [_execute_task(a, cache) for a in arg]

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in _execute_task

return func(*(_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py:82: in call

return read_parquet_part(

../../../.local/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py:490: in read_parquet_part

dfs = [

../../../.local/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py:491: in 

func(fs, rg, columns.copy(), index, **toolz.merge(kwargs, kw))

.tox/test-gpu/lib/python3.8/site-packages/merlin/io/parquet.py:128: in read_partition

return CudfEngine.read_partition(fs, pieces, *args, **kwargs)


raise MemoryError(

"Parquet data was larger than the available GPU memory!\n\n"

"See the notes on split_row_groups in the read_parquet "

"documentation.\n\n"

"Original Error: " + str(err)

)

E   MemoryError: Parquet data was larger than the available GPU memory!

E

E   See the notes on split_row_groups in the read_parquet documentation.

E

E   Original Error: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/parquet.py:265: MemoryError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:38,607 - distributed.worker - WARNING - Compute Failed

Key:       ('transform-6150d6df13aafcc4c59dcb4543fb4814', 1)

Function:  subgraph_callable-0481128e-7aa7-4063-9a5b-a7dff93a

args:      ({'piece': ('/tmp/pytest-of-jenkins/pytest-6/parquet0/dataset-1.parquet', [0], [])})

kwargs:    {}

Exception: "MemoryError('Parquet data was larger than the available GPU memory!\n\nSee the notes on split_row_groups in the read_parquet documentation.\n\nOriginal Error: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory')"
2022-12-01 17:41:38,608 - distributed.worker - WARNING - Compute Failed

Key:       ('transform-6150d6df13aafcc4c59dcb4543fb4814', 0)

Function:  subgraph_callable-0481128e-7aa7-4063-9a5b-a7dff93a

args:      ({'piece': ('/tmp/pytest-of-jenkins/pytest-6/parquet0/dataset-0.parquet', [0], [])})

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"
2022-12-01 17:41:38,614 - distributed.worker - WARNING - Compute Failed

Key:       ('transform-4653fb45d25fdf73b3d80938a29dcc58', 0)

Function:  subgraph_callable-534bba4d-2354-4006-b714-d219d6a3

args:      ({'piece': ('/tmp/pytest-of-jenkins/pytest-6/parquet0/dataset-0.parquet', [0], [])})

kwargs:    {}

Exception: "MemoryError('Parquet data was larger than the available GPU memory!\n\nSee the notes on split_row_groups in the read_parquet documentation.\n\nOriginal Error: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory')"
2022-12-01 17:41:38,621 - distributed.worker - WARNING - Compute Failed

Key:       ('transform-4653fb45d25fdf73b3d80938a29dcc58', 1)

Function:  subgraph_callable-534bba4d-2354-4006-b714-d219d6a3

args:      ({'piece': ('/tmp/pytest-of-jenkins/pytest-6/parquet0/dataset-1.parquet', [0], [])})

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/io/utilities/hostdevice_vector.hpp:57: 2 cudaErrorMemoryAllocation out of memory')"
___________________ test_gpu_workflow_api[True-True-csv-0.1] ___________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Tru3')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f28d46cf490>

gpu_memory_frac = 0.1, engine = 'csv', dump = True, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

/usr/local/lib/python3.8/dist-packages/dask_cudf/io/csv.py:149: in _read_csv

return cudf.read_csv(fn, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/io/csv.py:74: in read_csv

return libcudf.csv.read_csv(


???

E   MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory

cudf/_lib/csv.pyx:430: MemoryError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:41,462 - distributed.worker - WARNING - Compute Failed

Key:       ('read-csv-1de58f54ef56bb35810958a9255b04ab', 0)

Function:  _read_csv

args:      ('/tmp/pytest-of-jenkins/pytest-6/csv0/dataset-0.csv', array([dtype('O'), dtype('int64'), dtype('int64'), dtype('O'),

dtype('float64'), dtype('float64'), dtype('float64')], dtype=object))

kwargs:    {'storage_options': {}, 'byte_range': (0, 1706930995)}

Exception: "MemoryError('std::bad_alloc: out_of_memory: CUDA error at: /opt/rapids/rmm/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory')"
______________ test_gpu_workflow_api[True-True-csv-no-header-0.1] ______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Tru5')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f29568afa60>

gpu_memory_frac = 0.1, engine = 'csv-no-header', dump = True, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:45,337 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-39fae054ef745a3cac8d088f6bef9b33', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0  420.041901  438.897888  14920.920898, 'squaredsum':             x           y             id

0  207.678443  218.071076  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0  406.924896  429.870178  14927.633789, 'squaredsum':             x           y             id

0  195.741594  209.219389  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
________________ test_gpu_workflow_api[True-False-parquet-0.01] ________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Fal0')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c307ba580>

gpu_memory_frac = 0.01, engine = 'parquet', dump = False, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:46,417 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-da8a90e6be92389c9e5eb1871f2d1db1', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0  420.041901  438.897888  14920.920898, 'squaredsum':             x           y             id

0  207.678443  218.071076  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0  406.924896  429.870178  14927.633789, 'squaredsum':             x           y             id

0  195.741594  209.219389  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
________________ test_gpu_workflow_api[True-False-parquet-0.1] _________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Fal1')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2caa004280>

gpu_memory_frac = 0.1, engine = 'parquet', dump = False, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:47,748 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-3530d776a269eb58c1cef62c3fa8342b', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0  420.041901  438.897888  14920.920898, 'squaredsum':             x           y             id

0  207.678443  218.071076  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0  406.924896  429.870178  14927.633789, 'squaredsum':             x           y             id

0  195.741594  209.219389  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_____________ test_gpu_workflow_api[True-False-csv-no-header-0.01] _____________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Fal4')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f28c44cfe80>

gpu_memory_frac = 0.01, engine = 'csv-no-header', dump = False

use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:51,017 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-880646e56222f6c818f055a8f2377044', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0  420.041901  438.897888  14920.920898, 'squaredsum':             x           y             id

0  207.678443  218.071076  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0  406.924896  429.870178  14927.633789, 'squaredsum':             x           y             id

0  195.741594  209.219389  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_____________ test_gpu_workflow_api[True-False-csv-no-header-0.1] ______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_api_True_Fal5')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c30745d60>

gpu_memory_frac = 0.1, engine = 'csv-no-header', dump = False, use_client = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("use_client", [True, False])
def test_gpu_workflow_api(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, use_client):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    set_dask_client(client=client if use_client else None)
    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify(cat_cache="host")
    cont_features = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> ops.LogOp >> norms

    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:112:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:41:51,939 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-ba5b38b9fc029fe357b9e038c8b3ec49', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0  420.041901  438.897888  14920.920898, 'squaredsum':             x           y             id

0  207.678443  218.071076  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0  406.924896  429.870178  14927.633789, 'squaredsum':             x           y             id

0  195.741594  209.219389  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
________________________________ test_spec_set _________________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_spec_set0')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>
def test_spec_set(tmpdir, client):
    gdf_test = make_df(
        {
            "ad_id": [1, 2, 2, 6, 6, 8, 3, 3],
            "source_id": [2, 4, 4, 7, 5, 2, 5, 2],
            "platform": [1, 2, np.nan, 2, 1, 3, 3, 1],
            "cont": [1, 2, np.nan, 2, 1, 3, 3, 1],
            "clicked": [1, 0, 1, 0, 0, 1, 1, 0],
        }
    )

    cats = ColumnSelector(["ad_id", "source_id", "platform"])
    cat_features = cats >> ops.Categorify
    cont_features = ColumnSelector(["cont"]) >> ops.FillMissing >> ops.Normalize
    te_features = cats >> ops.TargetEncoding("clicked", kfold=5, fold_seed=42, p_smooth=20)

    set_dask_client(client=client)
    p = Workflow(cat_features + cont_features + te_features)


  p.fit_transform(nvt.Dataset(gdf_test)).to_ddf().compute()


tests/unit/workflow/test_workflow.py:194:

nvtabular/workflow/workflow.py:234: in fit_transform

self.fit(dataset)

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:92: in _finalize_moments

n = inp["count"].iloc[0]

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:138: in getitem

return self._getitem_tuple_arg(arg)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:456: in _getitem_tuple_arg

return self._downcast_to_series(df, arg)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:219: in _downcast_to_series

sr = df.T

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:37,634 - distributed.worker - WARNING - Compute Failed

Key:       global-moments-290e4c40024d638723f7e3d504203f22

Function:  _finalize_moments

args:      ({'count':    cont

0     8, 'sum':    cont

0  13.0, 'squaredsum':    cont

0  29.0})

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
2022-12-01 17:42:37,659 - distributed.worker - WARNING - Compute Failed

Key:       global-moments-536293ac8248693148f8360577879cd7

Function:  _finalize_moments

args:      ({'count':    clicked

0        8, 'sum':    clicked

0      4.0, 'squaredsum':    clicked

0      4.0})

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_____________________ test_gpu_workflow[True-parquet-0.01] _____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_True_parquet0')

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2ca8d81ac0>

gpu_memory_frac = 0.01, engine = 'parquet', dump = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
def test_gpu_workflow(tmpdir, df, dataset, gpu_memory_frac, engine, dump):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    conts = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> norms
    cats = cat_names >> ops.Categorify()
    workflow = nvt.Workflow(conts + cats + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:210:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:39,066 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-6d2fefddfab1dbda38272bb9079188d3', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x         y         id

0  545.692609  570.8883  2159119.0, 'squaredsum':             x           y            id

0  369.774444  388.684377  2.160456e+09}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y         id

0  524.859054  556.152692  2159847.0, 'squaredsum':             x           y            id

0  344.297198  370.008654  2.160758e+09}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_____________________ test_gpu_workflow[True-parquet-0.1] ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_True_parquet1')

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c841c7a00>

gpu_memory_frac = 0.1, engine = 'parquet', dump = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
def test_gpu_workflow(tmpdir, df, dataset, gpu_memory_frac, engine, dump):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    conts = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> norms
    cats = cat_names >> ops.Categorify()
    workflow = nvt.Workflow(conts + cats + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:210:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:40,100 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-904ee248547c4ac251208f3272a95ffc', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x         y         id

0  545.692609  570.8883  2159119.0, 'squaredsum':             x           y            id

0  369.774444  388.684377  2.160456e+09}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y         id

0  524.859054  556.152692  2159847.0, 'squaredsum':             x           y            id

0  344.297198  370.008654  2.160758e+09}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
__________________ test_gpu_workflow[True-csv-no-header-0.01] __________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_True_csv_no_0')

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f283eff48b0>

gpu_memory_frac = 0.01, engine = 'csv-no-header', dump = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
def test_gpu_workflow(tmpdir, df, dataset, gpu_memory_frac, engine, dump):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    conts = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> norms
    cats = cat_names >> ops.Categorify()
    workflow = nvt.Workflow(conts + cats + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:210:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:44,417 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-60c3584e68286268023d4828499e7fe1', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x         y         id

0  545.692609  570.8883  2159119.0, 'squaredsum':             x           y            id

0  369.774444  388.684377  2.160456e+09}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y         id

0  524.859054  556.152692  2159847.0, 'squaredsum':             x           y            id

0  344.297198  370.008654  2.160758e+09}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
__________________ test_gpu_workflow[True-csv-no-header-0.1] ___________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_True_csv_no_1')

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2ca8ced550>

gpu_memory_frac = 0.1, engine = 'csv-no-header', dump = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
def test_gpu_workflow(tmpdir, df, dataset, gpu_memory_frac, engine, dump):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    conts = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> norms
    cats = cat_names >> ops.Categorify()
    workflow = nvt.Workflow(conts + cats + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:210:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:45,220 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-9e47d47a8b4a3b154f63147a9cf14a10', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x         y         id

0  545.692609  570.8883  2159119.0, 'squaredsum':             x           y            id

0  369.774444  388.684377  2.160456e+09}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y         id

0  524.859054  556.152692  2159847.0, 'squaredsum':             x           y            id

0  344.297198  370.008654  2.160758e+09}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
____________________ test_gpu_workflow[False-parquet-0.01] _____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_False_parque0')

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c30731550>

gpu_memory_frac = 0.01, engine = 'parquet', dump = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
def test_gpu_workflow(tmpdir, df, dataset, gpu_memory_frac, engine, dump):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    conts = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> norms
    cats = cat_names >> ops.Categorify()
    workflow = nvt.Workflow(conts + cats + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:210:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:46,242 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-6f1df1f1e4d4b4cb5b4b4511e167e999', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x         y         id

0  545.692609  570.8883  2159119.0, 'squaredsum':             x           y            id

0  369.774444  388.684377  2.160456e+09}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y         id

0  524.859054  556.152692  2159847.0, 'squaredsum':             x           y            id

0  344.297198  370.008654  2.160758e+09}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_____________________ test_gpu_workflow[False-parquet-0.1] _____________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_False_parque1')

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c6c6d0670>

gpu_memory_frac = 0.1, engine = 'parquet', dump = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
def test_gpu_workflow(tmpdir, df, dataset, gpu_memory_frac, engine, dump):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    conts = cont_names >> ops.FillMissing() >> ops.Clip(min_value=0) >> norms
    cats = cat_names >> ops.Categorify()
    workflow = nvt.Workflow(conts + cats + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:210:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:47,214 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-64b076cf8b27634370d88171f80c8b6a', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x         y         id

0  545.692609  570.8883  2159119.0, 'squaredsum':             x           y            id

0  369.774444  388.684377  2.160456e+09}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y         id

0  524.859054  556.152692  2159847.0, 'squaredsum':             x           y            id

0  344.297198  370.008654  2.160758e+09}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_______________ test_gpu_workflow_config[True-True-parquet-0.01] _______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_True_0')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2ca978f310>

gpu_memory_frac = 0.01, engine = 'parquet', dump = True, replace = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:42:56,513 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-aa60655b91391d74b8289dabb01284bb', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0 -729.951294 -563.958984  14920.920898, 'squaredsum':              x           y             id

0  2541.541672  2091.07722  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0 -662.805664 -590.724854  14927.633789, 'squaredsum':              x            y             id

0  2350.272056  2154.738497  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
______________ test_gpu_workflow_config[True-False-parquet-0.01] _______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_True_6')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c6c7ee0d0>

gpu_memory_frac = 0.01, engine = 'parquet', dump = False, replace = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:07,438 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-e7ea6133ce4e60aa2fd0dfe9740a3f48', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0 -729.951294 -563.958984  14920.920898, 'squaredsum':              x           y             id

0  2541.541672  2091.07722  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0 -662.805664 -590.724854  14927.633789, 'squaredsum':              x            y             id

0  2350.272056  2154.738497  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
_______________ test_gpu_workflow_config[True-False-parquet-0.1] _______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_True_7')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c6c63ecd0>

gpu_memory_frac = 0.1, engine = 'parquet', dump = False, replace = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:08,615 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-fbae527831ffe5b8dfc372fce3546e01', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0 -729.951294 -563.958984  14920.920898, 'squaredsum':              x           y             id

0  2541.541672  2091.07722  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0 -662.805664 -590.724854  14927.633789, 'squaredsum':              x            y             id

0  2350.272056  2154.738497  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
___________ test_gpu_workflow_config[True-False-csv-no-header-0.01] ____________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_True_10')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2816b13280>

gpu_memory_frac = 0.01, engine = 'csv-no-header', dump = False, replace = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:12,841 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-7805d5f4bc9baf4635bf452a3e31336f', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0 -729.951294 -563.958984  14920.920898, 'squaredsum':              x            y             id

0  2541.541673  2091.077221  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0 -662.805664 -590.724854  14927.633789, 'squaredsum':              x            y             id

0  2350.272057  2154.738509  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
____________ test_gpu_workflow_config[True-False-csv-no-header-0.1] ____________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_True_11')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c6c6582b0>

gpu_memory_frac = 0.1, engine = 'csv-no-header', dump = False, replace = True
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:13,881 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-af0ef4b2d4c499324e0ab0d25e71f18d', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':       x     y    id

0  2160  2160  2160, 'sum':             x           y            id

0 -729.951294 -563.958984  14920.920898, 'squaredsum':              x            y             id

0  2541.541673  2091.077221  103073.463963}, {'count':       x     y    id

0  2161  2161  2161, 'sum':             x           y            id

0 -662.805664 -590.724854  14927.633789, 'squaredsum':              x            y             id

0  2350.272057  2154.738509  103118.338565}])

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
______________ test_gpu_workflow_config[False-True-parquet-0.01] _______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_False0')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2ca97408b0>

gpu_memory_frac = 0.01, engine = 'parquet', dump = True, replace = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:15,097 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-5a63de1d067e9698eeb5c9d7ef1e6e15', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2160                     2160  ...  2160  2160
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -729.951294              -563.958984  ...  55.893506  2159119.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y            id

0              2541.541672               2091.07722  ...  726.64947  2.160456e+09
[1 rows x 6 columns]}, {'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2161                     2161  ...  2161  2161
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -662.805664              -590.724854  ...  34.551734  2159847.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
____________ test_gpu_workflow_config[False-True-csv-no-header-0.1] ____________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_False5')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2816b58f40>

gpu_memory_frac = 0.1, engine = 'csv-no-header', dump = True, replace = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:25,358 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-d6c45c18cb97e9fc79aef09cd477f869', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2160                     2160  ...  2160  2160
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -729.951294              -563.958984  ...  55.893506  2159119.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y            id

0              2541.541673              2091.077221  ...  726.64947  2.160456e+09
[1 rows x 6 columns]}, {'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2161                     2161  ...  2161  2161
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -662.805664              -590.724854  ...  34.551734  2159847.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
______________ test_gpu_workflow_config[False-False-parquet-0.01] ______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_False6')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2ca8d98100>

gpu_memory_frac = 0.01, engine = 'parquet', dump = False, replace = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:26,406 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-ef70958d5b61a4cc5879b6f4080236ea', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2160                     2160  ...  2160  2160
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -729.951294              -563.958984  ...  55.893506  2159119.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y            id

0              2541.541672               2091.07722  ...  726.64947  2.160456e+09
[1 rows x 6 columns]}, {'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2161                     2161  ...  2161  2161
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -662.805664              -590.724854  ...  34.551734  2159847.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
______________ test_gpu_workflow_config[False-False-parquet-0.1] _______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_False7')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =       name-cat name-string    id  label         x         y

0          Dan         Tim  1045   1062 -0.145672 -0.89864...h   956    996 -0.716058 -0.292705

4320     Laura      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 6 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f2c145c6340>

gpu_memory_frac = 0.1, engine = 'parquet', dump = False, replace = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:27,678 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-189823fad5d442202cd2ab43ce2cffc0', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2160                     2160  ...  2160  2160
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -729.951294              -563.958984  ...  55.893506  2159119.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y            id

0              2541.541672               2091.07722  ...  726.64947  2.160456e+09
[1 rows x 6 columns]}, {'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2161                     2161  ...  2161  2161
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -662.805664              -590.724854  ...  34.551734  2159847.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
___________ test_gpu_workflow_config[False-False-csv-no-header-0.1] ____________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-6/test_gpu_workflow_config_False11')

client = <Client: 'tcp://127.0.0.1:43757' processes=2 threads=16, memory=125.83 GiB>

df =      name-string    id  label         x         y

0            Tim  1045   1062 -0.145672 -0.898646

1          Laura  ...     Hannah   956    996 -0.716058 -0.292705

2160      Yvonne  1015    975 -0.259790  0.710177
[4321 rows x 5 columns]

dataset = <merlin.io.dataset.Dataset object at 0x7f295459b5e0>

gpu_memory_frac = 0.1, engine = 'csv-no-header', dump = False, replace = False
@pytest.mark.parametrize("gpu_memory_frac", [0.01, 0.1])
@pytest.mark.parametrize("engine", ["parquet", "csv", "csv-no-header"])
@pytest.mark.parametrize("dump", [True, False])
@pytest.mark.parametrize("replace", [True, False])
def test_gpu_workflow_config(tmpdir, client, df, dataset, gpu_memory_frac, engine, dump, replace):
    cat_names = ["name-cat", "name-string"] if engine == "parquet" else ["name-string"]
    cont_names = ["x", "y", "id"]
    label_name = ["label"]

    norms = ops.Normalize()
    cat_features = cat_names >> ops.Categorify()
    if replace:
        cont_features = cont_names >> ops.FillMissing() >> ops.LogOp >> norms
    else:
        fillmissing_logop = (
            cont_names
            >> ops.FillMissing()
            >> ops.LogOp
            >> ops.Rename(postfix="_FillMissing_1_LogOp_1")
        )
        cont_features = cont_names + fillmissing_logop >> norms

    set_dask_client(client=client)
    workflow = Workflow(cat_features + cont_features + label_name)


  workflow.fit(dataset)


tests/unit/workflow/test_workflow.py:284:

nvtabular/workflow/workflow.py:198: in fit

self.executor.fit(ddf, current_phase)

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in fit

results = [r.result() for r in dask_client.compute(stats)]

.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py:334: in 

results = [r.result() for r in dask_client.compute(stats)]

../../../.local/lib/python3.8/site-packages/distributed/client.py:279: in result

raise exc.with_traceback(tb)

nvtabular/ops/moments.py:87: in _tree_node_moments

out[val] = _concat(df_list, ignore_index=True).sum().to_frame().transpose()

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:3313: in transpose

result_columns = libcudf.transpose.transpose(source_columns)


???

E   RuntimeError: parallel_for failed: cudaErrorMemoryAllocation: out of memory

cudf/_lib/transpose.pyx:21: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:43:35,288 - distributed.worker - WARNING - Compute Failed

Key:       ('chunkwise-moments-203dd45132e2d4b90511f6dbb9e9ad42', 0, 1)

Function:  _tree_node_moments

args:      ([{'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2160                     2160  ...  2160  2160
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -729.951294              -563.958984  ...  55.893506  2159119.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y            id

0              2541.541673              2091.077221  ...  726.64947  2.160456e+09
[1 rows x 6 columns]}, {'count':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...     y    id

0                     2161                     2161  ...  2161  2161
[1 rows x 6 columns], 'sum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...          y         id

0              -662.805664              -590.724854  ...  34.551734  2159847.0
[1 rows x 6 columns], 'squaredsum':    x_FillMissing_1_LogOp_1  y_FillMissing_1_LogOp_1  ...

kwargs:    {}

Exception: "RuntimeError('parallel_for failed: cudaErrorMemoryAllocation: out of memory')"
______________________ test_workflow_transform_ddf_dtypes ______________________
@pytest.mark.skipif(not cudf, reason="needs cudf")
def test_workflow_transform_ddf_dtypes():
    # Initial Dataset
    dtypes = {"name": str, "id": int, "x": float, "y": float}
    df = cudf.datasets.timeseries(dtypes=dtypes).reset_index()
    ddf = dask_cudf.from_cudf(df, npartitions=2)

    dataset = Dataset(ddf)

    # Create and Execute Workflow
    cols = ["name", "x", "y", "timestamp"]
    cat_cols = ["id"] >> ops.Normalize()
    workflow = Workflow(cols + cat_cols)
    workflow.fit(dataset)
    transformed_ddf = workflow.transform(dataset).to_ddf()

    # no transforms on the pass through cols, should have original dtypes
    for col in cols:
        assert_eq(ddf.dtypes[col], transformed_ddf.dtypes[col])

    # Followup dask-cudf sorting used to throw an exception because of dtype issues,
    # check that it works now


  transformed_ddf.sort_values(["id", "timestamp"]).compute()


tests/unit/workflow/test_workflow.py:631:

../../../.local/lib/python3.8/site-packages/dask/base.py:292: in compute

(result,) = compute(self, traverse=False, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/base.py:575: in compute

results = schedule(dsk, keys, **kwargs)

../../../.local/lib/python3.8/site-packages/distributed/client.py:3015: in get

results = self.gather(packed, asynchronous=asynchronous, direct=direct)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2167: in gather

return self.sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:309: in sync

return sync(

../../../.local/lib/python3.8/site-packages/distributed/utils.py:376: in sync

raise exc.with_traceback(tb)

../../../.local/lib/python3.8/site-packages/distributed/utils.py:349: in f

result = yield future

../../../.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762: in run

value = future.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:282: in __step

result = coro.throw(exc)

../../../.local/lib/python3.8/site-packages/distributed/client.py:2030: in _gather

raise exception.with_traceback(traceback)

../../../.local/lib/python3.8/site-packages/dask/optimization.py:990: in call

return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))

../../../.local/lib/python3.8/site-packages/dask/core.py:149: in get

result = _execute_task(task, cache)

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in _execute_task

return func((_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in 

return func((_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/core.py:119: in _execute_task

return func(*(_execute_task(a, cache) for a in args))

../../../.local/lib/python3.8/site-packages/dask/utils.py:39: in apply

return func(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/dask/dataframe/core.py:6330: in apply_and_enforce

df = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/nvtx/nvtx.py:101: in inner

result = func(*args, **kwargs)

/usr/local/lib/python3.8/dist-packages/dask_cudf/sorting.py:39: in _set_partitions_pre

partitions[s._columns[0].isnull().values] = (

/usr/local/lib/python3.8/dist-packages/cudf/core/column/column.py:622: in isnull

result = result | libcudf.unary.is_nan(self)

/usr/local/lib/python3.8/dist-packages/cudf/core/mixins/mixin_factory.py:11: in wrapper

return method(self, *args1, *args2, **kwargs1, **kwargs2)

/usr/local/lib/python3.8/dist-packages/cudf/core/column/numerical.py:223: in _binaryop

return libcudf.binaryop.binaryop(lhs, rhs, op, out_dtype)

cudf/_lib/binaryop.pyx:197: in cudf._lib.binaryop.binaryop

???


???

E   RuntimeError: CUDA error encountered at: ../src/binaryop/compiled/binary_ops.cuh:269: 98 cudaErrorInvalidDeviceFunction invalid device function

cudf/_lib/binaryop.pyx:109: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-12-01 17:44:09,288 - distributed.worker - WARNING - Compute Failed

Key:       ('assign-5fc754d9c541eb8e61051eab993ac742', 0)

Function:  subgraph_callable-403ef430-da1e-4595-95be-eca178d0

args:      (                  timestamp    id    name         x         y

0       2000-01-01 00:00:00   971     Ray -0.641726  0.149732

1       2000-01-01 00:00:01   960   Laura -0.369516  0.220369

2       2000-01-01 00:00:02   973   Quinn -0.443863  0.570721

3       2000-01-01 00:00:03   947   Laura -0.269333 -0.232744

4       2000-01-01 00:00:04   982   Sarah  0.151231  0.744703

...                     ...   ...     ...       ...       ...

1295996 2000-01-15 23:59:56   970   Wendy  0.168838  0.941327

1295997 2000-01-15 23:59:57  1034   Edith -0.139440 -0.795906

1295998 2000-01-15 23:59:58  1015   Alice -0.567523 -0.571742

1295999 2000-01-15 23:59:59   982   Alice -0.835511  0.195511

1296000 2000-01-16 00:00:00   972  Xavier  0.093860  0.295664
[1296001 rows x 5 columns])

kwargs:    {}

Exception: "RuntimeError('CUDA error encountered at: ../src/binaryop/compiled/binary_ops.cuh:269: 98 cudaErrorInvalidDeviceFunction invalid device function')"
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
tests/unit/test_dask_nvt.py: 6 warnings

tests/unit/workflow/test_workflow.py: 34 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/dask/base.py:1282: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.

warnings.warn(
tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py:431: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(
tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                Stmts   Miss  Cover
merlin/transforms/init.py           1      1     0%

merlin/transforms/ops/init.py       1      1     0%
TOTAL                                   2      2     0%
=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:14: could not import 'moto': No module named 'moto'

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

= 26 failed, 1392 passed, 2 skipped, 103 warnings, 20 errors in 1079.00s (0:17:59) =

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins9758627793138683739.sh

mikemckiernan · 2022-12-01T18:26:31Z

rerun tests

nvidia-merlin-bot · 2022-12-01T18:48:32Z

Click to view CI Results

GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
Running as SYSTEM
Setting status of 05c2b8c3737512cff6262a6a952ebfb2ec99f860 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/nvtabular_tests/4943/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on the built-in node in workspace /var/jenkins_home/jobs/nvtabular_tests/workspace
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1719/*:refs/remotes/origin/pr/1719/* # timeout=10
 > git rev-parse 05c2b8c3737512cff6262a6a952ebfb2ec99f860^{commit} # timeout=10
Checking out Revision 05c2b8c3737512cff6262a6a952ebfb2ec99f860 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 05c2b8c3737512cff6262a6a952ebfb2ec99f860 # timeout=10
Commit message: "Merge branch 'main' into dev-doc-link-testing"
 > git rev-list --no-walk 05c2b8c3737512cff6262a6a952ebfb2ec99f860 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins9462019059306658499.sh
GLOB sdist-make: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/setup.py
test-gpu recreate: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/.tmp/package/1/nvtabular-1.6.0+14.g05c2b8c3.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.3,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,-e git+https://github.com/NVIDIA-Merlin/NVTabular.git@05c2b8c3737512cff6262a6a952ebfb2ec99f860#egg=nvtabular,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1233574437'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-zz254e0n
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-zz254e0n
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=6cad626a63eafb4a98704f7067bd3552895df84de7dc04c24c17707d2a56ff07
  Stored in directory: /tmp/pip-ephem-wheel-cache-73baxvc4/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/dataloader.git
  Cloning https://github.com/NVIDIA-Merlin/dataloader.git to /tmp/pip-req-build-2j9v4k6u
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/dataloader.git /tmp/pip-req-build-2j9v4k6u
  Resolved https://github.com/NVIDIA-Merlin/dataloader.git to commit 498543d54629216dc09a5854dca4dbeeabcab356
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.8.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-dataloader==0.0.2+22.g498543d) (0.9.0+14.g4f73ff5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.0.1)
Building wheels for collected packages: merlin-dataloader
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2+22.g498543d-py3-none-any.whl size=40350 sha256=342ff7198876fe2f910254191a2a89a4532f08471dbaeb3004da25eedf4bef0d
  Stored in directory: /tmp/pip-ephem-wheel-cache-6ez6s_ak/wheels/de/f5/d9/251909f4627d2920fb15548f5ffd6daf1bf24c3c56bb4977b1
Successfully built merlin-dataloader
Installing collected packages: merlin-dataloader
  Attempting uninstall: merlin-dataloader
    Found existing installation: merlin-dataloader 0.0.3
    Uninstalling merlin-dataloader-0.0.3:
      Successfully uninstalled merlin-dataloader-0.0.3
Successfully installed merlin-dataloader-0.0.2+22.g498543d
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-wo_wa04s
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-wo_wa04s
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit e08a72c9c59416a9000e62d25548eb08367fc3fa
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.9.0+61.ge08a72c9) (0.9.0+14.g4f73ff5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.1)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+61.ge08a72c9-py3-none-any.whl size=367208 sha256=ccdbe2376054f7dda33385993eaccfa4a9372ca81b60dab8220e418fd95ce849
  Stored in directory: /tmp/pip-ephem-wheel-cache-wvl4yjyw/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+61.ge08a72c9
test-gpu run-test: commands[3] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 1439 items / 1 skipped
tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

....                                                                     [  8%]

tests/unit/test_notebooks.py ....                                        [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ................................     [ 12%]

tests/unit/examples/test_01-Getting-started.py .                         [ 12%]

tests/unit/examples/test_02-Advanced-NVTabular-workflow.py .             [ 12%]

tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py F      [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 23%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 25%]

.....................................................                    [ 29%]

tests/unit/ops/test_categorify.py ...................................... [ 31%]

........................................................................ [ 36%]

.....................................................                    [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 42%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .......................                  [ 47%]

tests/unit/ops/test_hash_bucket.py .........................             [ 49%]

tests/unit/ops/test_join.py ............................................ [ 52%]

........................................................................ [ 57%]

..................................                                       [ 59%]

tests/unit/ops/test_lambda.py ..........                                 [ 60%]

tests/unit/ops/test_normalize.py ....................................... [ 63%]

..                                                                       [ 63%]

tests/unit/ops/test_ops.py ............................................. [ 66%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 70%]

........................................................................ [ 75%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 88%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/ops/test_value_count.py ...                                   [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ................................... [ 92%]

..........................................................               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]
=================================== FAILURES ===================================

_______________________________ test_example_03 ________________________________
def test_example_03():
    with testbook(
        REPO_ROOT / "examples" / "03-Running-on-multiple-GPUs-or-on-CPU.ipynb",
        execute=False,
        timeout=180,
    ) as tb:
        tb.inject(
            """
            import os
            from unittest.mock import patch
            from merlin.datasets.synthetic import generate_data
            mock_train, mock_valid = generate_data(
                input="movielens-1m",
                num_rows=1000,
                set_sizes=(0.8, 0.2)
            )
            input_path = os.environ.get(
                "INPUT_DATA_DIR",
                os.path.expanduser("~/merlin-framework/movielens/")
            )
            from pathlib import Path
            Path(f'{input_path}ml-1m').mkdir(parents=True, exist_ok=True)
            mock_train.compute().to_parquet(f'{input_path}ml-1m/train.parquet')
            mock_train.compute().to_parquet(f'{input_path}ml-1m/valid.parquet')

            p1 = patch(
                "merlin.datasets.entertainment.get_movielens",
                return_value=[mock_train, mock_valid]
            )
            p1.start()

            """
        )


      tb.execute()


tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py:59:

../../../.local/lib/python3.8/site-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

../../../.local/lib/python3.8/site-packages/nbclient/util.py:84: in wrapped

return just_run(coro(*args, **kwargs))

../../../.local/lib/python3.8/site-packages/nbclient/util.py:62: in just_run

return loop.run_until_complete(coro)

/usr/local/lib/python3.8/dist-packages/nest_asyncio.py:89: in run_until_complete

return f.result()

/usr/lib/python3.8/asyncio/futures.py:178: in result

raise self._exception

/usr/lib/python3.8/asyncio/tasks.py:280: in __step

result = coro.send(None)

../../../.local/lib/python3.8/site-packages/nbclient/client.py:965: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)

self = <testbook.client.TestbookNotebookClient object at 0x7fc6968e2e80>

cell = {'cell_type': 'code', 'execution_count': 6, 'id': 'e02409ee', 'metadata': {'execution': {'iopub.status.busy': '2022-12...rkdir,\n        dashboard_address=":" + dashboard_port,\n        rmm_pool_size=(device_pool_size // 256) * 256\n    )'}

cell_index = 11

exec_reply = {'buffers': [], 'content': {'ename': 'MemoryError', 'engine_info': {'engine_id': -1, 'engine_uuid': '1ed995bb-65b1-4a0...e, 'engine': '1ed995bb-65b1-4a09-a48f-207ddab09899', 'started': '2022-12-01T18:37:18.942615Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E E           ------------------
 E           if cluster is None:
 E E E E E E E E E               )
 E           ------------------
 E
 E E E E           �[1;32m E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E
 E E           �[1;32m E           �[1;32m E           �[1;32m E           �[0;32m   (...)�[0m
 E           �[1;32m E           �[1;32m     40�[0m ):
 E           �[1;32m E           �[1;32m E           �[1;32m E           �[0;32m   (...)�[0m
 E           �[1;32m E           �[1;32m E E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E           �[1;32m E
 E E
 E E
 E E
 E E

nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
 cluster = LocalCUDACluster(
 protocol=protocol,
 n_workers=len(visible_devices.split(",")),
 CUDA_VISIBLE_DEVICES=visible_devices,
 device_memory_limit=device_limit,
 local_directory=dask_workdir,
 dashboard_address=":" + dashboard_port,
 rmm_pool_size=(device_pool_size // 256) * 256
 �[0;31m---------------------------------------------------------------------------�[0m
 �[0;31mMemoryError�[0m                               Traceback (most recent call last)
 Cell �[0;32mIn [6], line 2�[0m
 1�[0m �[38;5;28;01mif�[39;00m cluster �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m:
 �[0;32m----> 2�[0m     cluster �[38;5;241m=�[39m �[43mLocalCUDACluster�[49m�[43m(�[49m
 3�[0m �[43m        �[49m�[43mprotocol�[49m�[38;5;241;43m=�[39;49m�[43mprotocol�[49m�[43m,�[49m
 4�[0m �[43m        �[49m�[43mn_workers�[49m�[38;5;241;43m=�[39;49m�[38;5;28;43mlen�[39;49m�[43m(�[49m�[43mvisible_devices�[49m�[38;5;241;43m.�[39;49m�[43msplit�[49m�[43m(�[49m�[38;5;124;43m"�[39;49m�[38;5;124;43m,�[39;49m�[38;5;124;43m"�[39;49m�[43m)�[49m�[43m)�[49m�[43m,�[49m
 5�[0m �[43m        �[49m�[43mCUDA_VISIBLE_DEVICES�[49m�[38;5;241;43m=�[39;49m�[43mvisible_devices�[49m�[43m,�[49m
 6�[0m �[43m        �[49m�[43mdevice_memory_limit�[49m�[38;5;241;43m=�[39;49m�[43mdevice_limit�[49m�[43m,�[49m
 7�[0m �[43m        �[49m�[43mlocal_directory�[49m�[38;5;241;43m=�[39;49m�[43mdask_workdir�[49m�[43m,�[49m
 8�[0m �[43m        �[49m�[43mdashboard_address�[49m�[38;5;241;43m=�[39;49m�[38;5;124;43m"�[39;49m�[38;5;124;43m:�[39;49m�[38;5;124;43m"�[39;49m�[43m �[49m�[38;5;241;43m+�[39;49m�[43m �[49m�[43mdashboard_port�[49m�[43m,�[49m
 9�[0m �[43m        �[49m�[43mrmm_pool_size�[49m�[38;5;241;43m=�[39;49m�[43m(�[49m�[43mdevice_pool_size�[49m�[43m �[49m�[38;5;241;43m/�[39;49m�[38;5;241;43m/�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m�[43m)�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43m �[49m�[38;5;241;43m256�[39;49m
 10�[0m �[43m    �[49m�[43m)�[49m
 File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/local_cuda_cluster.py:367�[0m, in �[0;36mLocalCUDACluster.__init__�[0;34m(self, CUDA_VISIBLE_DEVICES, n_workers, threads_per_worker, memory_limit, device_memory_limit, data, local_directory, shared_filesystem, protocol, enable_tcp_over_ucx, enable_infiniband, enable_nvlink, enable_rdmacm, rmm_pool_size, rmm_maximum_pool_size, rmm_managed_memory, rmm_async, rmm_log_directory, rmm_track_allocations, jit_unspill, log_spilling, worker_class, pre_import, **kwargs)�[0m
 365�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mcuda_visible_devices �[38;5;241m=�[39m CUDA_VISIBLE_DEVICES
 366�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mscale(n_workers)
 �[0;32m--> 367�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43msync�[49m�[43m(�[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_correct_state�[49m�[43m)�[49m
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:309�[0m, in �[0;36mSyncMethodMixin.sync�[0;34m(self, func, asynchronous, callback_timeout, args, **kwargs)�[0m
 307�[0m     �[38;5;28;01mreturn�[39;00m future
 308�[0m �[38;5;28;01melse�[39;00m:
 �[0;32m--> 309�[0m     �[38;5;28;01mreturn�[39;00m �[43msync�[49m�[43m(�[49m
 310�[0m �[43m        �[49m�[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mloop�[49m�[43m,�[49m�[43m �[49m�[43mfunc�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[43margs�[49m�[43m,�[49m�[43m �[49m�[43mcallback_timeout�[49m�[38;5;241;43m=�[39;49m�[43mcallback_timeout�[49m�[43m,�[49m�[43m �[49m�[38;5;241;43m�[39;49m�[38;5;241;43m*�[39;49m�[43mkwargs�[49m
 311�[0m �[43m    �[49m�[43m)�[49m
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:376�[0m, in �[0;36msync�[0;34m(loop, func, callback_timeout, *args, **kwargs)�[0m
 374�[0m �[38;5;28;01mif�[39;00m error:
 375�[0m     typ, exc, tb �[38;5;241m=�[39m error
 �[0;32m--> 376�[0m     �[38;5;28;01mraise�[39;00m exc�[38;5;241m.�[39mwith_traceback(tb)
 377�[0m �[38;5;28;01melse�[39;00m:
 378�[0m     �[38;5;28;01mreturn�[39;00m result
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/utils.py:349�[0m, in �[0;36msync..f�[0;34m()�[0m
 347�[0m         future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mwait_for(future, callback_timeout)
 348�[0m     future �[38;5;241m=�[39m asyncio�[38;5;241m.�[39mensure_future(future)
 �[0;32m--> 349�[0m     result �[38;5;241m=�[39m �[38;5;28;01myield�[39;00m future
 350�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
 351�[0m     error �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
 File �[0;32m~/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg/tornado/gen.py:762�[0m, in �[0;36mRunner.run�[0;34m(self)�[0m
 759�[0m exc_info �[38;5;241m=�[39m �[38;5;28;01mNone�[39;00m
 761�[0m �[38;5;28;01mtry�[39;00m:
 �[0;32m--> 762�[0m     value �[38;5;241m=�[39m �[43mfuture�[49m�[38;5;241;43m.�[39;49m�[43mresult�[49m�[43m(�[49m�[43m)�[49m
 763�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
 764�[0m     exc_info �[38;5;241m=�[39m sys�[38;5;241m.�[39mexc_info()
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/deploy/spec.py:352�[0m, in �[0;36mSpecCluster._correct_state_internal�[0;34m(self)�[0m
 350�[0m     �[38;5;28;01mfor�[39;00m w �[38;5;129;01min�[39;00m workers:
 351�[0m         w�[38;5;241m.�[39m_cluster �[38;5;241m=�[39m weakref�[38;5;241m.�[39mref(�[38;5;28mself�[39m)
 �[0;32m--> 352�[0m         �[38;5;28;01mawait�[39;00m w  �[38;5;66;03m# for tornado gen.coroutine support�[39;00m
 353�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mworkers�[38;5;241m.�[39mupdate(�[38;5;28mdict�[39m(�[38;5;28mzip�[39m(to_open, workers)))
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36mServer.await.._�[0;34m()�[0m
 293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(
 294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(
 295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout
 296�[0m                 )
 297�[0m             )
 298�[0m     �[38;5;28;01melse�[39;00m:
 �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()
 300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning
 301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:347�[0m, in �[0;36mNanny.start�[0;34m(self)�[0m
 344�[0m     �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mplugin_add(plugin�[38;5;241m=�[39mplugin, name�[38;5;241m=�[39mname)
 346�[0m logger�[38;5;241m.�[39minfo(�[38;5;124m"�[39m�[38;5;124m        Start Nanny at: �[39m�[38;5;132;01m%r�[39;00m�[38;5;124m"�[39m, �[38;5;28mself�[39m�[38;5;241m.�[39maddress)
 �[0;32m--> 347�[0m response �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minstantiate()
 348�[0m �[38;5;28;01mif�[39;00m response �[38;5;241m==�[39m Status�[38;5;241m.�[39mrunning:
 349�[0m     �[38;5;28;01massert�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mworker_address
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:430�[0m, in �[0;36mNanny.instantiate�[0;34m(self)�[0m
 428�[0m �[38;5;28;01melse�[39;00m:
 429�[0m     �[38;5;28;01mtry�[39;00m:
 �[0;32m--> 430�[0m         result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mprocess�[38;5;241m.�[39mstart()
 431�[0m     �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
 432�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mclose()
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:685�[0m, in �[0;36mWorkerProcess.start�[0;34m(self)�[0m
 683�[0m     �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstatus
 684�[0m �[38;5;28;01mtry�[39;00m:
 �[0;32m--> 685�[0m     msg �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_wait_until_connected(uid)
 686�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m:
 687�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mfailed
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:803�[0m, in �[0;36mWorkerProcess._wait_until_connected�[0;34m(self, uid)�[0m
 799�[0m �[38;5;28;01mif�[39;00m �[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m �[38;5;129;01min�[39;00m msg:
 800�[0m     logger�[38;5;241m.�[39merror(
 801�[0m         �[38;5;124m"�[39m�[38;5;124mFailed while trying to start worker process: �[39m�[38;5;132;01m%s�[39;00m�[38;5;124m"�[39m, msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]
 802�[0m     )
 �[0;32m--> 803�[0m     �[38;5;28;01mraise�[39;00m msg[�[38;5;124m"�[39m�[38;5;124mexception�[39m�[38;5;124m"�[39m]
 804�[0m �[38;5;28;01melse�[39;00m:
 805�[0m     �[38;5;28;01mreturn�[39;00m msg
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/nanny.py:869�[0m, in �[0;36mrun�[0;34m()�[0m
 865�[0m �[38;5;124;03m"""�[39;00m
 866�[0m �[38;5;124;03mTry to start worker and inform parent of outcome.�[39;00m
 867�[0m �[38;5;124;03m"""�[39;00m
 868�[0m �[38;5;28;01mtry�[39;00m:
 �[0;32m--> 869�[0m     �[38;5;28;01mawait�[39;00m worker
 870�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:
 871�[0m     logger�[38;5;241m.�[39mexception(�[38;5;124m"�[39m�[38;5;124mFailed to start worker�[39m�[38;5;124m"�[39m)
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/core.py:299�[0m, in �[0;36m_�[0;34m()�[0m
 293�[0m             �[38;5;28;01mraise�[39;00m �[38;5;167;01mTimeoutError�[39;00m(
 294�[0m                 �[38;5;124m"�[39m�[38;5;132;01m{}�[39;00m�[38;5;124m failed to start in �[39m�[38;5;132;01m{}�[39;00m�[38;5;124m seconds�[39m�[38;5;124m"�[39m�[38;5;241m.�[39mformat(
 295�[0m                     �[38;5;28mtype�[39m(�[38;5;28mself�[39m)�[38;5;241m.�[39m�[38;5;18m__name__�[39m, timeout
 296�[0m                 )
 297�[0m             )
 298�[0m     �[38;5;28;01melse�[39;00m:
 �[0;32m--> 299�[0m         �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mstart()
 300�[0m         �[38;5;28mself�[39m�[38;5;241m.�[39mstatus �[38;5;241m=�[39m Status�[38;5;241m.�[39mrunning
 301�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:1372�[0m, in �[0;36mstart�[0;34m()�[0m
 1370�[0m         �[38;5;28;01mfor�[39;00m exc �[38;5;129;01min�[39;00m plugins_exceptions:
 1371�[0m             logger�[38;5;241m.�[39merror(�[38;5;28mrepr�[39m(exc))
 �[0;32m-> 1372�[0m     �[38;5;28;01mraise�[39;00m plugins_exceptions[�[38;5;241m0�[39m]
 1374�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_pending_plugins �[38;5;241m=�[39m ()
 1376�[0m �[38;5;28;01mawait�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39m_register_with_scheduler()
 File �[0;32m~/.local/lib/python3.8/site-packages/distributed/worker.py:3248�[0m, in �[0;36mplugin_add�[0;34m()�[0m
 3246�[0m �[38;5;28;01mif�[39;00m �[38;5;28mhasattr�[39m(plugin, �[38;5;124m"�[39m�[38;5;124msetup�[39m�[38;5;124m"�[39m):
 3247�[0m     �[38;5;28;01mtry�[39;00m:
 �[0;32m-> 3248�[0m         result �[38;5;241m=�[39m plugin�[38;5;241m.�[39msetup(worker�[38;5;241m=�[39m�[38;5;28mself�[39m)
 3249�[0m         �[38;5;28;01mif�[39;00m isawaitable(result):
 3250�[0m             result �[38;5;241m=�[39m �[38;5;28;01mawait�[39;00m result
 File �[0;32m/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py:78�[0m, in �[0;36msetup�[0;34m()�[0m
 74�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m
 76�[0m     pool_allocator �[38;5;241m=�[39m �[38;5;28;01mFalse�[39;00m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size �[38;5;129;01mis�[39;00m �[38;5;28;01mNone�[39;00m �[38;5;28;01melse�[39;00m �[38;5;28;01mTrue�[39;00m
 �[0;32m---> 78�[0m     rmm�[38;5;241m.�[39mreinitialize(
 79�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,
 80�[0m         managed_memory�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmanaged_memory,
 81�[0m         initial_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39minitial_pool_size,
 82�[0m         maximum_pool_size�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mmaximum_pool_size,
 83�[0m         logging�[38;5;241m=�[39m�[38;5;28mself�[39m�[38;5;241m.�[39mlogging,
 84�[0m         log_file_name�[38;5;241m=�[39mget_rmm_log_file_name(
 85�[0m             worker, �[38;5;28mself�[39m�[38;5;241m.�[39mlogging, �[38;5;28mself�[39m�[38;5;241m.�[39mlog_directory
 86�[0m         ),
 87�[0m     )
 88�[0m �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mrmm_track_allocations:
 89�[0m     �[38;5;28;01mimport�[39;00m �[38;5;21;01mrmm�[39;00m
 File �[0;32m/usr/local/lib/python3.8/dist-packages/rmm/rmm.py:85�[0m, in �[0;36mreinitialize�[0;34m()�[0m
 32�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mreinitialize�[39m(
 33�[0m     pool_allocator�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,
 34�[0m     managed_memory�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m,
 39�[0m     log_file_name�[38;5;241m=�[39m�[38;5;28;01mNone�[39;00m,
 41�[0m     �[38;5;124;03m"""�[39;00m
 42�[0m �[38;5;124;03m    Finalizes and then initializes RMM using the options passed. Using memory�[39;00m
 43�[0m �[38;5;124;03m    from a previous initialization of RMM is undefined behavior and should be�[39;00m
 83�[0m �[38;5;124;03m    corresponding to each device.�[39;00m
 84�[0m �[38;5;124;03m    """�[39;00m
 �[0;32m---> 85�[0m     rmm�[38;5;241m.�[39mmr�[38;5;241m.�[39m_initialize(
 86�[0m         pool_allocator�[38;5;241m=�[39mpool_allocator,
 87�[0m         managed_memory�[38;5;241m=�[39mmanaged_memory,
 88�[0m         initial_pool_size�[38;5;241m=�[39minitial_pool_size,
 89�[0m         maximum_pool_size�[38;5;241m=�[39mmaximum_pool_size,
 90�[0m         devices�[38;5;241m=�[39mdevices,
 91�[0m         logging�[38;5;241m=�[39mlogging,
 92�[0m         log_file_name�[38;5;241m=�[39mlog_file_name,
 93�[0m     )
 File �[0;32mmemory_resource.pyx:823�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m
 File �[0;32mmemory_resource.pyx:883�[0m, in �[0;36mrmm._lib.memory_resource._initialize�[0;34m()�[0m
 File �[0;32mmemory_resource.pyx:342�[0m, in �[0;36mrmm._lib.memory_resource.PoolMemoryResource.__cinit__�[0;34m()�[0m
 �[0;31mMemoryError�[0m: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
 MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded
../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-12-01 18:37:21,904 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:21,906 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:21,910 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:21,923 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:22,552 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:22,553 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:25,147 - distributed.diskutils - INFO - Found stale lock file and directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/test_dask/workdir/dask-worker-space/worker-snqrds8e', purging

2022-12-01 18:37:25,148 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:25,418 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:25,419 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,279 - distributed.diskutils - INFO - Found stale lock file and directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/test_dask/workdir/dask-worker-space/worker-7bo4qk7c', purging

2022-12-01 18:37:28,280 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:28,289 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:28,320 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:28,327 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:28,819 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,820 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,868 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,869 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,955 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,956 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,962 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:28,963 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:31,620 - distributed.diskutils - INFO - Found stale lock file and directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/test_dask/workdir/dask-worker-space/worker-b651i728', purging

2022-12-01 18:37:31,620 - distributed.diskutils - INFO - Found stale lock file and directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/test_dask/workdir/dask-worker-space/worker-d36aiyeg', purging

2022-12-01 18:37:31,620 - distributed.diskutils - INFO - Found stale lock file and directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/test_dask/workdir/dask-worker-space/worker-mja6qfpn', purging

2022-12-01 18:37:31,621 - distributed.diskutils - INFO - Found stale lock file and directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/test_dask/workdir/dask-worker-space/worker-lm25ccr2', purging

2022-12-01 18:37:31,621 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-12-01 18:37:31,892 - distributed.utils - ERROR - std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/utils.py", line 693, in log_errors

yield

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

2022-12-01 18:37:31,893 - distributed.nanny - ERROR - Failed to start worker

Traceback (most recent call last):

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/nanny.py", line 869, in run

await worker

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/core.py", line 299, in _

await self.start()

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 1372, in start

raise plugins_exceptions[0]

File "/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/worker.py", line 3248, in plugin_add

result = plugin.setup(worker=self)

File "/usr/local/lib/python3.8/dist-packages/dask_cuda/utils.py", line 78, in setup

rmm.reinitialize(

File "/usr/local/lib/python3.8/dist-packages/rmm/rmm.py", line 85, in reinitialize

rmm.mr._initialize(

File "memory_resource.pyx", line 823, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 883, in rmm._lib.memory_resource._initialize

File "memory_resource.pyx", line 342, in rmm._lib.memory_resource.PoolMemoryResource.cinit

MemoryError: std::bad_alloc: out_of_memory: RMM failure at:/opt/rapids/rmm/include/rmm/mr/device/pool_memory_resource.hpp:192: Maximum pool size exceeded

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 57 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
tests/unit/test_dask_nvt.py: 6 warnings

tests/unit/workflow/test_workflow.py: 78 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/dask/base.py:1282: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.

warnings.warn(
tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py:431: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(
tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                Stmts   Miss  Cover
merlin/transforms/init.py           1      1     0%

merlin/transforms/ops/init.py       1      1     0%
TOTAL                                   2      2     0%
=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:14: could not import 'moto': No module named 'moto'

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

===== 1 failed, 1437 passed, 2 skipped, 147 warnings in 1215.08s (0:20:15) =====

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

ERROR: InvocationError for command /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins10351210953376478539.sh

mikemckiernan · 2022-12-01T20:07:42Z

rerun tests

nvidia-merlin-bot · 2022-12-01T20:28:15Z

Click to view CI Results

GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
GitHub pull request #1719 of commit 05c2b8c3737512cff6262a6a952ebfb2ec99f860, no merge conflicts.
Running as SYSTEM
Setting status of 05c2b8c3737512cff6262a6a952ebfb2ec99f860 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/nvtabular_tests/4945/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on the built-in node in workspace /var/jenkins_home/jobs/nvtabular_tests/workspace
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1719/*:refs/remotes/origin/pr/1719/* # timeout=10
 > git rev-parse 05c2b8c3737512cff6262a6a952ebfb2ec99f860^{commit} # timeout=10
Checking out Revision 05c2b8c3737512cff6262a6a952ebfb2ec99f860 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 05c2b8c3737512cff6262a6a952ebfb2ec99f860 # timeout=10
Commit message: "Merge branch 'main' into dev-doc-link-testing"
 > git rev-list --no-walk 1da6f6323fa07ffbf50b0b482340e4c2f4067346 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins6989947506957112190.sh
GLOB sdist-make: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/setup.py
test-gpu recreate: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/.tmp/package/1/nvtabular-1.6.0+14.g05c2b8c3.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.3,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,-e git+https://github.com/NVIDIA-Merlin/NVTabular.git@05c2b8c3737512cff6262a6a952ebfb2ec99f860#egg=nvtabular,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3441507980'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-2d99hk3u
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-2d99hk3u
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=3ff03c9f7cd963a677c261c4c899c788a1e6afcabe7dcdf697551ff17b808841
  Stored in directory: /tmp/pip-ephem-wheel-cache-a9kusj5x/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
test-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/dataloader.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/dataloader.git
  Cloning https://github.com/NVIDIA-Merlin/dataloader.git to /tmp/pip-req-build-e2fcj9eb
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/dataloader.git /tmp/pip-req-build-e2fcj9eb
  Resolved https://github.com/NVIDIA-Merlin/dataloader.git to commit 498543d54629216dc09a5854dca4dbeeabcab356
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.8.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-dataloader==0.0.2+22.g498543d) (0.9.0+14.g4f73ff5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.8.0->merlin-dataloader==0.0.2+22.g498543d) (4.0.0)
Building wheels for collected packages: merlin-dataloader
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.2+22.g498543d-py3-none-any.whl size=40350 sha256=4922b2269d7e4f8abbab5c641007577f905f2062374bbe5c44175f959e9d2e81
  Stored in directory: /tmp/pip-ephem-wheel-cache-qd8q2n10/wheels/de/f5/d9/251909f4627d2920fb15548f5ffd6daf1bf24c3c56bb4977b1
Successfully built merlin-dataloader
Installing collected packages: merlin-dataloader
  Attempting uninstall: merlin-dataloader
    Found existing installation: merlin-dataloader 0.0.3
    Uninstalling merlin-dataloader-0.0.3:
      Successfully uninstalled merlin-dataloader-0.0.3
Successfully installed merlin-dataloader-0.0.2+22.g498543d
test-gpu run-test: commands[2] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/models.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/models.git
  Cloning https://github.com/NVIDIA-Merlin/models.git to /tmp/pip-req-build-6p0t0bhh
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/models.git /tmp/pip-req-build-6p0t0bhh
  Resolved https://github.com/NVIDIA-Merlin/models.git to commit e08a72c9c59416a9000e62d25548eb08367fc3fa
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-models==0.9.0+61.ge08a72c9) (0.9.0+14.g4f73ff5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.4.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-models==0.9.0+61.ge08a72c9) (4.0.0)
Building wheels for collected packages: merlin-models
  Building wheel for merlin-models (pyproject.toml): started
  Building wheel for merlin-models (pyproject.toml): finished with status 'done'
  Created wheel for merlin-models: filename=merlin_models-0.9.0+61.ge08a72c9-py3-none-any.whl size=367208 sha256=62a9634c12ea84f489126f1e859ebefa18ee528dc1806ae9c321f6be600ee296
  Stored in directory: /tmp/pip-ephem-wheel-cache-8uc52c4s/wheels/5a/43/99/d50fe2c33b4f4686db73207ce3865e0d6be6609ffb03abade5
Successfully built merlin-models
Installing collected packages: merlin-models
  Attempting uninstall: merlin-models
    Found existing installation: merlin-models 0.7.0+11.g280956aa4
    Not uninstalling merlin-models at /usr/local/lib/python3.8/dist-packages, outside environment /var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu
    Can't uninstall 'merlin-models'. No files were found to uninstall.
Successfully installed merlin-models-0.9.0+61.ge08a72c9
test-gpu run-test: commands[3] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 1439 items / 1 skipped
tests/unit/test_dask_nvt.py ............................................ [  3%]

........................................................................ [  8%]

....                                                                     [  8%]

tests/unit/test_notebooks.py ....                                        [  8%]

tests/unit/test_tf4rec.py .                                              [  8%]

tests/unit/test_tools.py ......................                          [ 10%]

tests/unit/test_triton_inference.py ................................     [ 12%]

tests/unit/examples/test_01-Getting-started.py .                         [ 12%]

tests/unit/examples/test_02-Advanced-NVTabular-workflow.py .             [ 12%]

tests/unit/examples/test_03-Running-on-multiple-GPUs-or-on-CPU.py .      [ 12%]

tests/unit/framework_utils/test_tf_feature_columns.py .                  [ 12%]

tests/unit/framework_utils/test_tf_layers.py ........................... [ 14%]

...................................................                      [ 18%]

tests/unit/framework_utils/test_torch_layers.py .                        [ 18%]

tests/unit/loader/test_tf_dataloader.py ................................ [ 20%]

........................................s..                              [ 23%]

tests/unit/loader/test_torch_dataloader.py ............................. [ 25%]

.....................................................                    [ 29%]

tests/unit/ops/test_categorify.py ...................................... [ 31%]

........................................................................ [ 36%]

.....................................................                    [ 40%]

tests/unit/ops/test_column_similarity.py ........................        [ 42%]

tests/unit/ops/test_drop_low_cardinality.py ..                           [ 42%]

tests/unit/ops/test_fill.py ............................................ [ 45%]

........                                                                 [ 45%]

tests/unit/ops/test_groupyby.py .......................                  [ 47%]

tests/unit/ops/test_hash_bucket.py .........................             [ 49%]

tests/unit/ops/test_join.py ............................................ [ 52%]

........................................................................ [ 57%]

..................................                                       [ 59%]

tests/unit/ops/test_lambda.py ..........                                 [ 60%]

tests/unit/ops/test_normalize.py ....................................... [ 63%]

..                                                                       [ 63%]

tests/unit/ops/test_ops.py ............................................. [ 66%]

....................                                                     [ 67%]

tests/unit/ops/test_ops_schema.py ...................................... [ 70%]

........................................................................ [ 75%]

........................................................................ [ 80%]

........................................................................ [ 85%]

.......................................                                  [ 88%]

tests/unit/ops/test_reduce_dtype_size.py ..                              [ 88%]

tests/unit/ops/test_target_encode.py .....................               [ 89%]

tests/unit/ops/test_value_count.py ...                                   [ 89%]

tests/unit/workflow/test_cpu_workflow.py ......                          [ 90%]

tests/unit/workflow/test_workflow.py ................................... [ 92%]

..........................................................               [ 96%]

tests/unit/workflow/test_workflow_chaining.py ...                        [ 96%]

tests/unit/workflow/test_workflow_node.py ...........                    [ 97%]

tests/unit/workflow/test_workflow_ops.py ...                             [ 97%]

tests/unit/workflow/test_workflow_schemas.py ........................... [ 99%]

...                                                                      [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
tests/unit/test_dask_nvt.py: 6 warnings

tests/unit/workflow/test_workflow.py: 78 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/dask/base.py:1282: UserWarning: Running on a single-machine scheduler when a distributed client is active might lead to unexpected results.

warnings.warn(
tests/unit/test_dask_nvt.py::test_merlin_core_execution_managers

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/utils.py:431: UserWarning: Existing Dask-client object detected in the current context. New cuda cluster will not be deployed. Set force_new to True to ignore running clusters.

warnings.warn(
tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]

tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]

tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/ops/test_ops_schema.py: 12 warnings

/var/jenkins_home/workspace/nvtabular_tests/nvtabular/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                Stmts   Miss  Cover
merlin/transforms/init.py           1      1     0%

merlin/transforms/ops/init.py       1      1     0%
TOTAL                                   2      2     0%
=========================== short test summary info ============================

SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:14: could not import 'moto': No module named 'moto'

SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:529: not working correctly in ci environment

========== 1438 passed, 2 skipped, 147 warnings in 1127.07s (0:18:47) ==========

/usr/local/lib/python3.8/dist-packages/coverage/control.py:801: CoverageWarning: No data was collected. (no-data-collected)

self._warn("No data was collected.", slug="no-data-collected")

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins5108190111148325489.sh

mikemckiernan · 2022-12-01T20:30:31Z

I know that I suggested waiting until another PR was merged, but when Jenkins finally says you can merge, you really, really need to merge.

* Migrate the legacy examples to the Merlin repo We may (or may not) want to keep these examples but they've overstayed their welcome in the NVTabular repo, which is burdened with the accumulation of a lot of historical cruft. Since some of these examples use inference code that's moving to Systems, it makes more sense for them to live in the Merlin repo (if we want to keep them.) * update READMEs * docs: Contribute to examples clean up - Fix difficult to detect broken links. - Revise TOC. * Handle data loader as an iterator (#1720) * Update test_gpu_dl_break to handle data loader as an iterator * Use peek method to look at first batch in notebooks * Revert whitespace change to image cell * Revert change to PyTorch training example notebook * Call peek on data iter to get batch * Describe how to check for broken links (#1719) This is one way to check for broken links, but I'm happy to adopt something that is better. Co-authored-by: Karl Higley <[email protected]> Co-authored-by: Benedikt Schifferer <[email protected]> Co-authored-by: Mike McKiernan <[email protected]> Co-authored-by: Oliver Holworthy <[email protected]>

Describe how to check for broken links

0f3608c

This is one way to check for broken links, but I'm happy to adopt something that is better.

mikemckiernan added skip-changelog Exclude from change log chore labels Nov 29, 2022

mikemckiernan requested a review from karlhigley November 29, 2022 17:43

mikemckiernan self-assigned this Nov 29, 2022

mikemckiernan requested a review from bschifferer November 29, 2022 17:44

karlhigley approved these changes Dec 1, 2022

View reviewed changes

Merge branch 'main' into dev-doc-link-testing

05c2b8c

mikemckiernan merged commit 51af616 into NVIDIA-Merlin:main Dec 1, 2022

mikemckiernan deleted the dev-doc-link-testing branch December 1, 2022 20:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Describe how to check for broken links #1719

Describe how to check for broken links #1719

mikemckiernan commented Nov 29, 2022

mikemckiernan commented Nov 29, 2022

nvidia-merlin-bot commented Nov 29, 2022

2022-11-29 17:54:42,054 - distributed.worker - WARNING - Compute Failed
Key: ('level_2-3fe3f94678ad7cf6cb6240303ad14b12', 2, 3)
Function: _mid_level_groupby
args: ([pyarrow.Table
zipcode: int32
zipcode_size: int32

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Nov 30, 2022

nvidia-merlin-bot commented Nov 30, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

github-actions bot commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

mikemckiernan commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Dec 1, 2022

Describe how to check for broken links #1719

Describe how to check for broken links #1719

Conversation

mikemckiernan commented Nov 29, 2022

mikemckiernan commented Nov 29, 2022

nvidia-merlin-bot commented Nov 29, 2022

2022-11-29 17:54:42,054 - distributed.worker - WARNING - Compute Failed Key: ('level_2-3fe3f94678ad7cf6cb6240303ad14b12', 2, 3) Function: _mid_level_groupby args: ([pyarrow.Table zipcode: int32 zipcode_size: int32

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0% merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Nov 30, 2022

nvidia-merlin-bot commented Nov 30, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0% merlin/transforms/ops/init.py 1 1 0%

github-actions bot commented Dec 1, 2022

Documentation preview

nvidia-merlin-bot commented Dec 1, 2022

mikemckiernan commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0% merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0% merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0% merlin/transforms/ops/init.py 1 1 0%

mikemckiernan commented Dec 1, 2022

2022-11-29 17:54:42,054 - distributed.worker - WARNING - Compute Failed
Key: ('level_2-3fe3f94678ad7cf6cb6240303ad14b12', 2, 3)
Function: _mid_level_groupby
args: ([pyarrow.Table
zipcode: int32
zipcode_size: int32

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

merlin/transforms/init.py 1 1 0%
merlin/transforms/ops/init.py 1 1 0%