Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Jenkinsfile #230

Merged
merged 3 commits into from
Nov 22, 2022
Merged

Add Jenkinsfile #230

merged 3 commits into from
Nov 22, 2022

Conversation

AyodeAwe
Copy link
Contributor

This adds the Jenkinsfile pipeline that will implement the PR builds to be run on gpuci.io's spawned cloud resources.

@AyodeAwe AyodeAwe added the gpuCI-test-this Enables PR to be discovered and tested on gpuci.io label Nov 16, 2022
@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/systems/review/pr-230

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #230 of commit 6d0f27231090960b4e3c2d245b3760fdce1477ec, no merge conflicts.
Running as SYSTEM
Setting status of 6d0f27231090960b4e3c2d245b3760fdce1477ec to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/512/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/230/*:refs/remotes/origin/pr/230/* # timeout=10
 > git rev-parse 6d0f27231090960b4e3c2d245b3760fdce1477ec^{commit} # timeout=10
Checking out Revision 6d0f27231090960b4e3c2d245b3760fdce1477ec (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6d0f27231090960b4e3c2d245b3760fdce1477ec # timeout=10
Commit message: "add jenkinsfile"
 > git rev-list --no-walk 522ffe75026f8feddb7549b2bb9642940b4915f4 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins8101487228167711722.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+15.g6d0f272.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.10,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.10,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+15.g6d0f272,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='695844807'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-odh6wzrd
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-odh6wzrd
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit aac1dc33cf92cd19f5a065fa7f49cf9b7500cfb2
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (3.19.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+12.gaac1dc3) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+12.gaac1dc3-py3-none-any.whl size=118622 sha256=8ea58709d10126af215f035f689148826f26a9bedba92668de7a138988f9d946
  Stored in directory: /tmp/pip-ephem-wheel-cache-scvgv0hs/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+12.gaac1dc3
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+15.g6d0f272) (1.1.1)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+15.g6d0f272) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+15.g6d0f272) (0.8.0+12.gaac1dc3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (3.19.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.g6d0f272) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+15.g6d0f272-py3-none-any.whl size=102099 sha256=5597b4d0238ee1a22d9dd1bb74d51cf6871db404025af6dc445822026507e973
  Stored in directory: /tmp/pip-ephem-wheel-cache-519n5id6/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+15.g6d0f272
    Uninstalling merlin-systems-0.7.0+15.g6d0f272:
      Successfully uninstalled merlin-systems-0.7.0+15.g6d0f272
Successfully installed merlin-systems-0.7.0+15.g6d0f272
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 91 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 12%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 27%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 39%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py .FF...                           [100%]


=================================== FAILURES ===================================

______________________________ test_torch_backend ______________________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-10/test_torch_backend0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_torch_backend(tmpdir):
    model_repository = Path(tmpdir)

    model_dir = model_repository / model_name
    model_version_dir = model_dir / "1"
    model_version_dir.mkdir(parents=True, exist_ok=True)

    # Write config out
    config_path = model_dir / "config.pbtxt"
    with open(str(config_path), "w", encoding="utf-8") as f:
        f.write(model_config)

    # Write model
    model_scripted = torch.jit.script(model)
    model_scripted.save(str(model_version_dir / "model.pt"))

    input_data = {"input": np.array([[2.0, 3.0, 4.0], [4.0, 8.0, 1.0]]).astype(np.float32)}

    inputs = [
        grpcclient.InferInput(
            "input", input_data["input"].shape, triton.np_to_triton_dtype(input_data["input"].dtype)
        )
    ]
    inputs[0].set_data_from_numpy(input_data["input"])

    outputs = [grpcclient.InferRequestedOutput("OUTPUT__0")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/torch/test_op.py:148:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-10/test_torch_backend0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1116 17:01:15.326749 30181 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fc764000000' with size 268435456

I1116 17:01:15.327487 30181 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1116 17:01:15.336507 30181 model_lifecycle.cc:459] loading: example_model:1

I1116 17:01:15.742697 30181 libtorch.cc:1983] TRITONBACKEND_Initialize: pytorch

I1116 17:01:15.742739 30181 libtorch.cc:1993] Triton TRITONBACKEND API version: 1.10

I1116 17:01:15.742745 30181 libtorch.cc:1999] 'pytorch' TRITONBACKEND API version: 1.10

I1116 17:01:15.742767 30181 libtorch.cc:2032] TRITONBACKEND_ModelInitialize: example_model (version 1)

W1116 17:01:15.743270 30181 libtorch.cc:284] skipping model configuration auto-complete for 'example_model': not supported for pytorch backend

I1116 17:01:15.744009 30181 libtorch.cc:313] Optimized execution is enabled for model instance 'example_model'

I1116 17:01:15.744030 30181 libtorch.cc:332] Cache Cleaning is disabled for model instance 'example_model'

I1116 17:01:15.744038 30181 libtorch.cc:349] Inference Mode is enabled for model instance 'example_model'

I1116 17:01:15.744045 30181 libtorch.cc:444] NvFuser is not specified for model instance 'example_model'

I1116 17:01:15.744382 30181 libtorch.cc:2076] TRITONBACKEND_ModelInstanceInitialize: example_model (GPU device 0)

I1116 17:01:15.832053 30181 libtorch.cc:2110] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1116 17:01:15.832105 30181 libtorch.cc:2055] TRITONBACKEND_ModelFinalize: delete model state

E1116 17:01:15.832121 30181 model_lifecycle.cc:596] failed to load 'example_model' version 1: Internal: failed to load model 'example_model': CUDA error: out of memory

CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.

For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Exception raised from currentStreamCaptureStatusMayInitCtx at /opt/pytorch/pytorch/c10/cuda/CUDAGraphsC10Utils.h:83 (most recent call first):

frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x6c (0x7fc7a05176dc in /opt/tritonserver/backends/pytorch/libc10.so)

frame #1:  + 0x236e4 (0x7fc7a049e6e4 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)

frame #2:  + 0x241e3 (0x7fc7a049f1e3 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)

frame #3:  + 0x44ca2 (0x7fc7a04bfca2 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)

frame #4:  + 0x44f68 (0x7fc7a04bff68 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)

frame #5: at::detail::empty_strided_generic(c10::ArrayRef, c10::ArrayRef, c10::Allocator*, c10::DispatchKeySet, c10::ScalarType) + 0x59f (0x7fc74290aacf in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #6: at::detail::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::ScalarType, c10::optionalc10::Device) + 0x9c (0x7fc7158d5e4c in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)

frame #7: at::detail::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x52 (0x7fc7158d5fe2 in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)

frame #8: at::native::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x4b (0x7fc7159d0c1b in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)

frame #9:  + 0x2dfe77f (0x7fc7178db77f in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)

frame #10:  + 0x2dfe8a4 (0x7fc7178db8a4 in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)

frame #11: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x102 (0x7fc7434e4d02 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #12:  + 0x1f8a7eb (0x7fc7437b07eb in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #13: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x1c8 (0x7fc743528838 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #14:  + 0x130435c (0x7fc742b2a35c in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #15: at::native::_to_copy(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x1749 (0x7fc742e8cf59 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #16:  + 0x216d43f (0x7fc74399343f in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #17: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0xf9 (0x7fc743246dc9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #18:  + 0x1f8cfd3 (0x7fc7437b2fd3 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #19: at::_ops::_to_copy::call(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x209 (0x7fc7432be6c9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #20: at::native::to(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optionalc10::MemoryFormat) + 0xfc (0x7fc742e832fc in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #21:  + 0x22c620d (0x7fc743aec20d in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #22: at::_ops::to_device::call(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optionalc10::MemoryFormat) + 0x1cd (0x7fc743409dcd in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #23: torch::jit::Unpickler::readInstruction() + 0x255 (0x7fc745a33e75 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #24: torch::jit::Unpickler::run() + 0xa8 (0x7fc745a36468 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #25: torch::jit::Unpickler::parse_ivalue() + 0x32 (0x7fc745a36612 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #26: torch::jit::readArchiveAndTensors(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, c10::optional<std::function<c10::StrongTypePtr (c10::QualifiedName const&)> >, c10::optional<std::function<c10::intrusive_ptr<c10::ivalue::Object, c10::detail::intrusive_target_default_null_typec10::ivalue::Object > (c10::StrongTypePtr, c10::IValue)> >, c10::optionalc10::Device, caffe2::serialize::PyTorchStreamReader&, c10::Type::SingletonOrSharedTypePtrc10::Type (*)(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&), std::shared_ptrtorch::jit::DeserializationStorageContext) + 0x569 (0x7fc7459f38e9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #27:  + 0x41b7579 (0x7fc7459dd579 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #28:  + 0x41ba6a6 (0x7fc7459e06a6 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #29: torch::jit::import_ir_module(std::shared_ptrtorch::jit::CompilationUnit, std::istream&, c10::optionalc10::Device, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > >&) + 0x179 (0x7fc7459e1b99 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #30: torch::jit::import_ir_module(std::shared_ptrtorch::jit::CompilationUnit, std::istream&, c10::optionalc10::Device) + 0x8f (0x7fc7459e1e8f in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #31: torch::jit::load(std::istream&, c10::optionalc10::Device) + 0xb9 (0x7fc7459e1f99 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)

frame #32:  + 0x1ee41 (0x7fc7a05e4e41 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)

frame #33:  + 0x247eb (0x7fc7a05ea7eb in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)

frame #34:  + 0x24c92 (0x7fc7a05eac92 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)

frame #35: TRITONBACKEND_ModelInstanceInitialize + 0x3f6 (0x7fc7a05eb0d6 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)

frame #36:  + 0x10cfb2 (0x7fc7acbe6fb2 in /opt/tritonserver/lib/libtritonserver.so)

frame #37:  + 0x10e732 (0x7fc7acbe8732 in /opt/tritonserver/lib/libtritonserver.so)

frame #38:  + 0x10288f (0x7fc7acbdc88f in /opt/tritonserver/lib/libtritonserver.so)

frame #39:  + 0x1bc3a4 (0x7fc7acc963a4 in /opt/tritonserver/lib/libtritonserver.so)

frame #40:  + 0x1c2e38 (0x7fc7acc9ce38 in /opt/tritonserver/lib/libtritonserver.so)

frame #41:  + 0x2f5b00 (0x7fc7acdcfb00 in /opt/tritonserver/lib/libtritonserver.so)

frame #42:  + 0xd6de4 (0x7fc7ac727de4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)

frame #43:  + 0x8609 (0x7fc7ada78609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)

frame #44: clone + 0x43 (0x7fc7ac414133 in /usr/lib/x86_64-linux-gnu/libc.so.6)


I1116 17:01:15.832288 30181 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1116 17:01:15.832380 30181 server.cc:588]

+---------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                    | Config                                                                                                                                                        |

+---------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| pytorch | /opt/tritonserver/backends/pytorch/libtriton_pytorch.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 17:01:15.832718 30181 server.cc:631]

+---------------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model         | Version | Status                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

+---------------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| example_model | 1       | UNAVAILABLE: Internal: failed to load model 'example_model': CUDA error: out of memory                                                                                                                                                                                                                                                                                                                                                                                                |

|               |         | CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.                                                                                                                                                                                                                                                                                                                                                                |

|               |         | For debugging consider passing CUDA_LAUNCH_BLOCKING=1.                                                                                                                                                                                                                                                                                                                                                                                                                                |

|               |         | Exception raised from currentStreamCaptureStatusMayInitCtx at /opt/pytorch/pytorch/c10/cuda/CUDAGraphsC10Utils.h:83 (most recent call first):                                                                                                                                                                                                                                                                                                                                         |

|               |         | frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x6c (0x7fc7a05176dc in /opt/tritonserver/backends/pytorch/libc10.so)                                                                                                                                                                                                                                                                             |

|               |         | frame #1:  + 0x236e4 (0x7fc7a049e6e4 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)                                                                                                                                                                                                                                                                                                                                                                          |

|               |         | frame #2:  + 0x241e3 (0x7fc7a049f1e3 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)                                                                                                                                                                                                                                                                                                                                                                          |

|               |         | frame #3:  + 0x44ca2 (0x7fc7a04bfca2 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)                                                                                                                                                                                                                                                                                                                                                                          |

|               |         | frame #4:  + 0x44f68 (0x7fc7a04bff68 in /opt/tritonserver/backends/pytorch/libc10_cuda.so)                                                                                                                                                                                                                                                                                                                                                                          |

|               |         | frame #5: at::detail::empty_strided_generic(c10::ArrayRef, c10::ArrayRef, c10::Allocator*, c10::DispatchKeySet, c10::ScalarType) + 0x59f (0x7fc74290aacf in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                           |

|               |         | frame #6: at::detail::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::ScalarType, c10::optionalc10::Device) + 0x9c (0x7fc7158d5e4c in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)                                                                                                                                                                                                                                                                        |

|               |         | frame #7: at::detail::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x52 (0x7fc7158d5fe2 in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)                                                                                                                                                                                                        |

|               |         | frame #8: at::native::empty_strided_cuda(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x4b (0x7fc7159d0c1b in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)                                                                                                                                                                                                        |

|               |         | frame #9:  + 0x2dfe77f (0x7fc7178db77f in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #10:  + 0x2dfe8a4 (0x7fc7178db8a4 in /opt/tritonserver/backends/pytorch/libtorch_cuda.so)                                                                                                                                                                                                                                                                                                                                                                     |

|               |         | frame #11: at::_ops::empty_strided::redispatch(c10::DispatchKeySet, c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x102 (0x7fc7434e4d02 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                             |

|               |         | frame #12:  + 0x1f8a7eb (0x7fc7437b07eb in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #13: at::_ops::empty_strided::call(c10::ArrayRef, c10::ArrayRef, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional) + 0x1c8 (0x7fc743528838 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                        |

|               |         | frame #14:  + 0x130435c (0x7fc742b2a35c in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #15: at::native::_to_copy(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x1749 (0x7fc742e8cf59 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                               |

|               |         | frame #16:  + 0x216d43f (0x7fc74399343f in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #17: at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0xf9 (0x7fc743246dc9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                  |

|               |         | frame #18:  + 0x1f8cfd3 (0x7fc7437b2fd3 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #19: at::_ops::_to_copy::call(at::Tensor const&, c10::optionalc10::ScalarType, c10::optionalc10::Layout, c10::optionalc10::Device, c10::optional, bool, c10::optionalc10::MemoryFormat) + 0x209 (0x7fc7432be6c9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                            |

|               |         | frame #20: at::native::to(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optionalc10::MemoryFormat) + 0xfc (0x7fc742e832fc in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                |

|               |         | frame #21:  + 0x22c620d (0x7fc743aec20d in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #22: at::_ops::to_device::call(at::Tensor const&, c10::Device, c10::ScalarType, bool, bool, c10::optionalc10::MemoryFormat) + 0x1cd (0x7fc743409dcd in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                    |

|               |         | frame #23: torch::jit::Unpickler::readInstruction() + 0x255 (0x7fc745a33e75 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                    |

|               |         | frame #24: torch::jit::Unpickler::run() + 0xa8 (0x7fc745a36468 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #25: torch::jit::Unpickler::parse_ivalue() + 0x32 (0x7fc745a36612 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                        |

|               |         | frame #26: torch::jit::readArchiveAndTensors(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, c10::optional<std::function<c10::StrongTypePtr (c10::QualifiedName const&)> >, c10::optional<std::function<c10::intrusive_ptr<c10::ivalue::Object, c10::detail:: |

|               |         | intrusive_target_default_null_typec10::ivalue::Object > (c10::StrongTypePtr, c10::IValue)> >, c10::optionalc10::Device, caffe2::serialize::PyTorchStreamReader&, c10::Type::SingletonOrSharedTypePtrc10::Type (*)(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&), std::shared_ptrtorch::jit::DeserializationStorageContext) + 0x569 (0x7fc7459f38e9 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                           |

|               |         | frame #27:  + 0x41b7579 (0x7fc7459dd579 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                                                      |

|               |         | frame #29: torch::jit::import_ir_module(std::shared_ptrtorch::jit::CompilationUnit, std::istream&, c10::optionalc10::Device, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<ch |

|               |         | ar>, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > >&) + 0x179 (0x7fc7459e1b99 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                 |

|               |         | frame #29: torch::jit::import_ir_module(std::shared_ptrtorch::jit::CompilationUnit, std::istream&, c10::optionalc10::Device, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::hash<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator > const, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > >&) + 0x179 (0x7fc7459e1b99 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so) |

|               |         | frame #30: torch::jit::import_ir_module(std::shared_ptrtorch::jit::CompilationUnit, std::istream&, c10::optionalc10::Device) + 0x8f (0x7fc7459e1e8f in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                        |

|               |         | frame #31: torch::jit::load(std::istream&, c10::optionalc10::Device) + 0xb9 (0x7fc7459e1f99 in /opt/tritonserver/backends/pytorch/libtorch_cpu.so)                                                                                                                                                                                                                                                                                                                                  |

|               |         | frame #32:  + 0x1ee41 (0x7fc7a05e4e41 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)                                                                                                                                                                                                                                                                                                                                                                   |

|               |         | frame #33:  + 0x247eb (0x7fc7a05ea7eb in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)                                                                                                                                                                                                                                                                                                                                                                   |

|               |         | frame #34:  + 0x24c92 (0x7fc7a05eac92 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)                                                                                                                                                                                                                                                                                                                                                                   |

|               |         | frame #35: TRITONBACKEND_ModelInstanceInitialize + 0x3f6 (0x7fc7a05eb0d6 in /opt/tritonserver/backends/pytorch/libtriton_pytorch.so)                                                                                                                                                                                                                                                                                                                                                  |

|               |         | frame #36:  + 0x10cfb2 (0x7fc7acbe6fb2 in /opt/tritonserver/lib/libtritonserver.so)                                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #37:  + 0x10e732 (0x7fc7acbe8732 in /opt/tritonserver/lib/libtritonserver.so)                                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #38:  + 0x10288f (0x7fc7acbdc88f in /opt/tritonserver/lib/libtritonserver.so)                                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #39:  + 0x1bc3a4 (0x7fc7acc963a4 in /opt/tritonserver/lib/libtritonserver.so)                                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #40:  + 0x1c2e38 (0x7fc7acc9ce38 in /opt/tritonserver/lib/libtritonserver.so)                                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #41:  + 0x2f5b00 (0x7fc7acdcfb00 in /opt/tritonserver/lib/libtritonserver.so)                                                                                                                                                                                                                                                                                                                                                                                 |

|               |         | frame #42:  + 0xd6de4 (0x7fc7ac727de4 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)                                                                                                                                                                                                                                                                                                                                                                                  |

|               |         | frame #43:  + 0x8609 (0x7fc7ada78609 in /usr/lib/x86_64-linux-gnu/libpthread.so.0)                                                                                                                                                                                                                                                                                                                                                                                  |

+---------------+---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 17:01:15.895524 30181 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1116 17:01:15.896414 30181 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-10/test_torch_backend0                                                                                                                                         |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 17:01:15.896449 30181 server.cc:262] Waiting for in-flight requests to complete.

I1116 17:01:15.896456 30181 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1116 17:01:15.896462 30181 server.cc:293] All models are stopped, unloading models

I1116 17:01:15.896467 30181 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1116 17:01:16.925216 30181 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

______________________ test_pytorch_op_serving[True-True] ______________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-10/test_pytorch_op_serving_True_T0')

use_path = True, torchscript = True


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("torchscript", [True])
@pytest.mark.parametrize("use_path", [True, False])
def test_pytorch_op_serving(tmpdir, use_path, torchscript):
    model_name = "0_predictpytorch"
    model_path = str(tmpdir / "model.pt")

    model_to_use = model_scripted if torchscript else model
    model_or_path = model_path if use_path else model_to_use

    if use_path:
        try:
            # jit-compiled version of a model
            model_to_use.save(model_path)
        except AttributeError:
            # non-jit-compiled version of a model
            torch.save(model_to_use, model_path)

    predictions = ["input"] >> ptorch_op.PredictPyTorch(
        model_or_path, model_input_schema, model_output_schema
    )
    ensemble = Ensemble(predictions, model_input_schema)
    ens_config, node_configs = ensemble.export(tmpdir)

    input_data = {"input": np.array([[2.0, 3.0, 4.0], [4.0, 8.0, 1.0]]).astype(np.float32)}

    inputs = [
        grpcclient.InferInput(
            "input", input_data["input"].shape, triton.np_to_triton_dtype(input_data["input"].dtype)
        )
    ]
    inputs[0].set_data_from_numpy(input_data["input"])

    outputs = [grpcclient.InferRequestedOutput("OUTPUT__0")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/torch/test_op.py:190:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-10/test_pytorch_op_serving_True_T0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

W1116 17:01:19.058154 30261 pinned_memory_manager.cc:236] Unable to allocate pinned system memory, pinned memory pool will not be available: out of memory

I1116 17:01:19.058929 30261 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1116 17:01:19.188259 30261 server.cc:209] Failed to finalize CUDA memory manager: CNMEM_STATUS_OUT_OF_MEMORY

I1116 17:01:19.191098 30261 model_lifecycle.cc:459] loading: 0_predictpytorch:1

I1116 17:01:19.615590 30261 libtorch.cc:1983] TRITONBACKEND_Initialize: pytorch

I1116 17:01:19.615622 30261 libtorch.cc:1993] Triton TRITONBACKEND API version: 1.10

I1116 17:01:19.615629 30261 libtorch.cc:1999] 'pytorch' TRITONBACKEND API version: 1.10

I1116 17:01:19.615650 30261 libtorch.cc:2032] TRITONBACKEND_ModelInitialize: 0_predictpytorch (version 1)

W1116 17:01:19.616168 30261 libtorch.cc:284] skipping model configuration auto-complete for '0_predictpytorch': not supported for pytorch backend

I1116 17:01:19.616898 30261 libtorch.cc:313] Optimized execution is enabled for model instance '0_predictpytorch'

I1116 17:01:19.616911 30261 libtorch.cc:332] Cache Cleaning is disabled for model instance '0_predictpytorch'

I1116 17:01:19.616918 30261 libtorch.cc:349] Inference Mode is enabled for model instance '0_predictpytorch'

I1116 17:01:19.616925 30261 libtorch.cc:444] NvFuser is not specified for model instance '0_predictpytorch'

I1116 17:01:19.617281 30261 libtorch.cc:2076] TRITONBACKEND_ModelInstanceInitialize: 0_predictpytorch (GPU device 0)

I1116 17:01:19.742557 30261 libtorch.cc:2110] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1116 17:01:19.742601 30261 libtorch.cc:2055] TRITONBACKEND_ModelFinalize: delete model state

E1116 17:01:19.742619 30261 model_lifecycle.cc:596] failed to load '0_predictpytorch' version 1: Internal: unable to create stream: out of memory

E1116 17:01:19.742685 30261 model_repository_manager.cc:481] Invalid argument: ensemble 'ensemble_model' depends on '0_predictpytorch' which has no loaded version

I1116 17:01:19.742776 30261 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1116 17:01:19.742851 30261 server.cc:588]

+---------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                    | Config                                                                                                                                                        |

+---------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| pytorch | /opt/tritonserver/backends/pytorch/libtriton_pytorch.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 17:01:19.742914 30261 server.cc:631]

+------------------+---------+---------------------------------------------------------------+

| Model            | Version | Status                                                        |

+------------------+---------+---------------------------------------------------------------+

| 0_predictpytorch | 1       | UNAVAILABLE: Internal: unable to create stream: out of memory |

+------------------+---------+---------------------------------------------------------------+


I1116 17:01:19.808751 30261 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1116 17:01:19.809584 30261 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-10/test_pytorch_op_serving_True_T0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 17:01:19.809620 30261 server.cc:262] Waiting for in-flight requests to complete.

I1116 17:01:19.809627 30261 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1116 17:01:19.809635 30261 server.cc:293] All models are stopped, unloading models

I1116 17:01:19.809648 30261 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1116 17:01:20.832384 30261 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            222     31    86%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         82      5    94%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        65     19    71%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      54      5    91%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2749   1184    57%


============ 2 failed, 89 passed, 55 warnings in 1239.95s (0:20:39) ============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins5476322535788170972.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #230 of commit d29050e60cf3b49c0077b6dcbd7228d6fa57ffd5, no merge conflicts.
Running as SYSTEM
Setting status of d29050e60cf3b49c0077b6dcbd7228d6fa57ffd5 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/514/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/230/*:refs/remotes/origin/pr/230/* # timeout=10
 > git rev-parse d29050e60cf3b49c0077b6dcbd7228d6fa57ffd5^{commit} # timeout=10
Checking out Revision d29050e60cf3b49c0077b6dcbd7228d6fa57ffd5 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d29050e60cf3b49c0077b6dcbd7228d6fa57ffd5 # timeout=10
Commit message: "add jenkinsfile"
 > git rev-list --no-walk f259208059b7a2e3d432cdb81e75fc1902d5155f # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins7734170090984597854.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+15.gd29050e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.10,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.10,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+15.gd29050e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='257681766'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ekeas5fk
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ekeas5fk
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit aac1dc33cf92cd19f5a065fa7f49cf9b7500cfb2
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (3.19.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+12.gaac1dc3) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+12.gaac1dc3-py3-none-any.whl size=118622 sha256=828e255d1a211d39d3c6d933df654cee2b461b7ff81c7c27b24d35efb55fd92d
  Stored in directory: /tmp/pip-ephem-wheel-cache-sj0zjtfm/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+12.gaac1dc3
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+15.gd29050e) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+15.gd29050e) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+15.gd29050e) (0.8.0+12.gaac1dc3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.2.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (3.19.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.gd29050e) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+15.gd29050e-py3-none-any.whl size=102098 sha256=ac3affdad244b91aa9b631b3a912c55e2a3e983850925cfeef22b2be00484886
  Stored in directory: /tmp/pip-ephem-wheel-cache-p_tr5hpl/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+15.gd29050e
    Uninstalling merlin-systems-0.7.0+15.gd29050e:
      Successfully uninstalled merlin-systems-0.7.0+15.gd29050e
Successfully installed merlin-systems-0.7.0+15.gd29050e
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 91 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 12%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 27%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 39%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            222     31    86%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         82      5    94%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        65     19    71%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      54      5    91%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     14    81%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2749   1186    57%


================= 91 passed, 55 warnings in 1200.93s (0:20:00) =================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins18296842316911754807.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #230 of commit e463c25927025a573c75eaf0c8075ef91b1757b0, no merge conflicts.
Running as SYSTEM
Setting status of e463c25927025a573c75eaf0c8075ef91b1757b0 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/515/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/230/*:refs/remotes/origin/pr/230/* # timeout=10
 > git rev-parse e463c25927025a573c75eaf0c8075ef91b1757b0^{commit} # timeout=10
Checking out Revision e463c25927025a573c75eaf0c8075ef91b1757b0 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e463c25927025a573c75eaf0c8075ef91b1757b0 # timeout=10
Commit message: "add jenkinsfile"
 > git rev-list --no-walk d29050e60cf3b49c0077b6dcbd7228d6fa57ffd5 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins12581546572311570455.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+15.ge463c25.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.10,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.10,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+15.ge463c25,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@ba4c14159a8e858c8998d4158a4376e65a8fa266,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3227565320'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-2b2hebch
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-2b2hebch
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit aac1dc33cf92cd19f5a065fa7f49cf9b7500cfb2
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+12.gaac1dc3) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (0.55.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+12.gaac1dc3) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+12.gaac1dc3) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+12.gaac1dc3) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+12.gaac1dc3) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+12.gaac1dc3) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+12.gaac1dc3) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+12.gaac1dc3-py3-none-any.whl size=118622 sha256=6317561ec2290dd61074e2533c1a1a70fb4e66bb9964148ef6cdc465f0069956
  Stored in directory: /tmp/pip-ephem-wheel-cache-bek53izw/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+12.gaac1dc3
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+15.ge463c25) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+15.ge463c25) (0.8.0+12.gaac1dc3)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+15.ge463c25) (1.1.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (0.55.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+15.ge463c25) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+15.ge463c25-py3-none-any.whl size=102098 sha256=a54b8193e8af566aed5ba57fc4b0e5cdb084fa42b08a7bc21deaf4923966701c
  Stored in directory: /tmp/pip-ephem-wheel-cache-pcjknioh/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+15.ge463c25
    Uninstalling merlin-systems-0.7.0+15.ge463c25:
      Successfully uninstalled merlin-systems-0.7.0+15.ge463c25
Successfully installed merlin-systems-0.7.0+15.ge463c25
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 91 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 12%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 27%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 39%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py ....F....                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_____________ test_ensemble[BayesianPersonalizedRanking-runtime1] ______________


model_cls = <function BayesianPersonalizedRanking at 0x7f20a1a64940>

runtime = <merlin.systems.dag.runtimes.triton.runtime.TritonExecutorRuntime object at 0x7f20973e1190>

tmpdir = local('/tmp/pytest-of-jenkins/pytest-21/test_ensemble_BayesianPersonal1')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("runtime", [None, TritonExecutorRuntime()])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, runtime, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema)
    ensemble_config, _ = triton_ens.export(tmpdir, runtime=runtime)

    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/implicit/test_op.py:121:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-21/test_ensemble_BayesianPersonal1')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------


0%|          | 0/100 [00:00<?, ?it/s]

1%|          | 1/100 [00:00<00:00, 788.40it/s, train_auc=50.00%, skipped=10.57%]

2%|▏         | 2/100 [00:00<00:00, 937.80it/s, train_auc=49.72%, skipped=11.08%]

3%|▎         | 3/100 [00:00<00:00, 1009.14it/s, train_auc=50.06%, skipped=12.49%]

4%|▍         | 4/100 [00:00<00:00, 1049.89it/s, train_auc=48.74%, skipped=11.98%]

5%|▌         | 5/100 [00:00<00:00, 1075.52it/s, train_auc=51.07%, skipped=10.47%]

6%|▌         | 6/100 [00:00<00:00, 1091.13it/s, train_auc=48.48%, skipped=13.60%]

7%|▋         | 7/100 [00:00<00:00, 1104.26it/s, train_auc=50.23%, skipped=11.98%]

8%|▊         | 8/100 [00:00<00:00, 1115.21it/s, train_auc=50.17%, skipped=10.47%]

9%|▉         | 9/100 [00:00<00:00, 1123.24it/s, train_auc=51.14%, skipped=11.58%]

10%|█         | 10/100 [00:00<00:00, 1128.93it/s, train_auc=50.17%, skipped=11.48%]

11%|█         | 11/100 [00:00<00:00, 1133.85it/s, train_auc=52.10%, skipped=11.08%]

12%|█▏        | 12/100 [00:00<00:00, 1124.13it/s, train_auc=51.51%, skipped=13.39%]

13%|█▎        | 13/100 [00:00<00:00, 1109.69it/s, train_auc=48.62%, skipped=12.19%]

14%|█▍        | 14/100 [00:00<00:00, 1097.68it/s, train_auc=50.00%, skipped=10.78%]

15%|█▌        | 15/100 [00:00<00:00, 1095.25it/s, train_auc=51.19%, skipped=11.08%]

16%|█▌        | 16/100 [00:00<00:00, 1104.82it/s, train_auc=49.02%, skipped=12.69%]

17%|█▋        | 17/100 [00:00<00:00, 1113.47it/s, train_auc=52.43%, skipped=10.88%]

18%|█▊        | 18/100 [00:00<00:00, 1120.86it/s, train_auc=47.54%, skipped=11.88%]

19%|█▉        | 19/100 [00:00<00:00, 1125.76it/s, train_auc=47.97%, skipped=13.29%]

20%|██        | 20/100 [00:00<00:00, 1131.56it/s, train_auc=50.74%, skipped=11.08%]

21%|██        | 21/100 [00:00<00:00, 1136.64it/s, train_auc=51.07%, skipped=10.27%]

22%|██▏       | 22/100 [00:00<00:00, 1141.12it/s, train_auc=51.52%, skipped=10.47%]

23%|██▎       | 23/100 [00:00<00:00, 1145.98it/s, train_auc=47.73%, skipped=11.18%]

24%|██▍       | 24/100 [00:00<00:00, 1146.96it/s, train_auc=51.81%, skipped=10.78%]

25%|██▌       | 25/100 [00:00<00:00, 1151.04it/s, train_auc=48.99%, skipped=10.57%]

26%|██▌       | 26/100 [00:00<00:00, 1155.20it/s, train_auc=48.06%, skipped=11.78%]

27%|██▋       | 27/100 [00:00<00:00, 1140.12it/s, train_auc=50.56%, skipped=10.78%]

28%|██▊       | 28/100 [00:00<00:00, 1142.01it/s, train_auc=51.76%, skipped=11.28%]

29%|██▉       | 29/100 [00:00<00:00, 1130.65it/s, train_auc=50.80%, skipped=11.58%]

30%|███       | 30/100 [00:00<00:00, 1132.61it/s, train_auc=49.78%, skipped=9.37%]

31%|███       | 31/100 [00:00<00:00, 1100.30it/s, train_auc=51.50%, skipped=12.59%]

32%|███▏      | 32/100 [00:00<00:00, 1080.68it/s, train_auc=49.60%, skipped=10.88%]

33%|███▎      | 33/100 [00:00<00:00, 1079.97it/s, train_auc=51.10%, skipped=12.69%]

34%|███▍      | 34/100 [00:00<00:00, 1084.81it/s, train_auc=52.10%, skipped=11.28%]

35%|███▌      | 35/100 [00:00<00:00, 1089.46it/s, train_auc=51.50%, skipped=12.59%]

36%|███▌      | 36/100 [00:00<00:00, 1092.66it/s, train_auc=50.74%, skipped=11.48%]

37%|███▋      | 37/100 [00:00<00:00, 1096.84it/s, train_auc=48.20%, skipped=10.57%]

38%|███▊      | 38/100 [00:00<00:00, 1100.94it/s, train_auc=51.74%, skipped=10.47%]

39%|███▉      | 39/100 [00:00<00:00, 1104.60it/s, train_auc=47.84%, skipped=11.38%]

40%|████      | 40/100 [00:00<00:00, 1108.30it/s, train_auc=49.43%, skipped=10.98%]

41%|████      | 41/100 [00:00<00:00, 1111.47it/s, train_auc=50.45%, skipped=10.37%]

42%|████▏     | 42/100 [00:00<00:00, 1114.62it/s, train_auc=49.89%, skipped=11.18%]

43%|████▎     | 43/100 [00:00<00:00, 1117.73it/s, train_auc=49.43%, skipped=12.39%]

44%|████▍     | 44/100 [00:00<00:00, 1120.90it/s, train_auc=48.70%, skipped=10.67%]

45%|████▌     | 45/100 [00:00<00:00, 1119.31it/s, train_auc=48.74%, skipped=12.39%]

46%|████▌     | 46/100 [00:00<00:00, 1121.50it/s, train_auc=45.52%, skipped=10.17%]

47%|████▋     | 47/100 [00:00<00:00, 1121.31it/s, train_auc=52.57%, skipped=9.77%]

48%|████▊     | 48/100 [00:00<00:00, 1124.05it/s, train_auc=51.24%, skipped=10.78%]

49%|████▉     | 49/100 [00:00<00:00, 1126.81it/s, train_auc=51.53%, skipped=10.88%]

50%|█████     | 50/100 [00:00<00:00, 1129.47it/s, train_auc=49.44%, skipped=10.37%]

51%|█████     | 51/100 [00:00<00:00, 1132.01it/s, train_auc=50.22%, skipped=10.17%]

52%|█████▏    | 52/100 [00:00<00:00, 1134.34it/s, train_auc=51.97%, skipped=12.99%]

53%|█████▎    | 53/100 [00:00<00:00, 1036.10it/s, train_auc=49.15%, skipped=11.08%]

54%|█████▍    | 54/100 [00:00<00:00, 1032.49it/s, train_auc=51.25%, skipped=11.18%]

55%|█████▌    | 55/100 [00:00<00:00, 1035.88it/s, train_auc=52.45%, skipped=11.48%]

56%|█████▌    | 56/100 [00:00<00:00, 1039.34it/s, train_auc=50.80%, skipped=12.39%]

57%|█████▋    | 57/100 [00:00<00:00, 1042.79it/s, train_auc=51.99%, skipped=11.28%]

58%|█████▊    | 58/100 [00:00<00:00, 1046.10it/s, train_auc=51.35%, skipped=10.37%]

59%|█████▉    | 59/100 [00:00<00:00, 1049.26it/s, train_auc=50.57%, skipped=11.78%]

60%|██████    | 60/100 [00:00<00:00, 1052.31it/s, train_auc=49.10%, skipped=10.98%]

61%|██████    | 61/100 [00:00<00:00, 1055.31it/s, train_auc=49.03%, skipped=11.88%]

62%|██████▏   | 62/100 [00:00<00:00, 1058.25it/s, train_auc=51.86%, skipped=10.67%]

63%|██████▎   | 63/100 [00:00<00:00, 1061.10it/s, train_auc=52.56%, skipped=11.48%]

64%|██████▍   | 64/100 [00:00<00:00, 1063.86it/s, train_auc=50.96%, skipped=10.67%]

65%|██████▌   | 65/100 [00:00<00:00, 1066.03it/s, train_auc=49.77%, skipped=11.98%]

66%|██████▌   | 66/100 [00:00<00:00, 1068.62it/s, train_auc=53.73%, skipped=10.98%]

67%|██████▋   | 67/100 [00:00<00:00, 1071.24it/s, train_auc=47.85%, skipped=13.29%]

68%|██████▊   | 68/100 [00:00<00:00, 1073.81it/s, train_auc=49.49%, skipped=11.68%]

69%|██████▉   | 69/100 [00:00<00:00, 1004.87it/s, train_auc=47.79%, skipped=11.08%]

70%|███████   | 70/100 [00:00<00:00, 1004.44it/s, train_auc=50.80%, skipped=12.19%]

71%|███████   | 71/100 [00:00<00:00, 1006.48it/s, train_auc=48.24%, skipped=11.08%]

72%|███████▏  | 72/100 [00:00<00:00, 1009.50it/s, train_auc=54.35%, skipped=11.98%]

73%|███████▎  | 73/100 [00:00<00:00, 1012.40it/s, train_auc=50.80%, skipped=12.39%]

74%|███████▍  | 74/100 [00:00<00:00, 1015.24it/s, train_auc=51.46%, skipped=10.17%]

75%|███████▌  | 75/100 [00:00<00:00, 1016.46it/s, train_auc=49.49%, skipped=11.68%]

76%|███████▌  | 76/100 [00:00<00:00, 1019.26it/s, train_auc=46.34%, skipped=10.67%]

77%|███████▋  | 77/100 [00:00<00:00, 1021.97it/s, train_auc=48.80%, skipped=11.88%]

78%|███████▊  | 78/100 [00:00<00:00, 1024.66it/s, train_auc=49.49%, skipped=10.88%]

79%|███████▉  | 79/100 [00:00<00:00, 1027.25it/s, train_auc=50.17%, skipped=11.28%]

80%|████████  | 80/100 [00:00<00:00, 1027.78it/s, train_auc=46.51%, skipped=13.39%]

81%|████████  | 81/100 [00:00<00:00, 1028.05it/s, train_auc=49.66%, skipped=10.57%]

82%|████████▏ | 82/100 [00:00<00:00, 1028.28it/s, train_auc=50.28%, skipped=10.67%]

83%|████████▎ | 83/100 [00:00<00:00, 1028.31it/s, train_auc=51.62%, skipped=10.07%]

84%|████████▍ | 84/100 [00:00<00:00, 1026.24it/s, train_auc=49.04%, skipped=11.08%]

85%|████████▌ | 85/100 [00:00<00:00, 1026.42it/s, train_auc=48.37%, skipped=10.67%]

86%|████████▌ | 86/100 [00:00<00:00, 1026.65it/s, train_auc=50.74%, skipped=12.08%]

87%|████████▋ | 87/100 [00:00<00:00, 1028.29it/s, train_auc=51.60%, skipped=11.98%]

88%|████████▊ | 88/100 [00:00<00:00, 1029.49it/s, train_auc=50.23%, skipped=10.98%]

89%|████████▉ | 89/100 [00:00<00:00, 1031.73it/s, train_auc=55.03%, skipped=10.88%]

90%|█████████ | 90/100 [00:00<00:00, 1034.04it/s, train_auc=49.02%, skipped=12.69%]

91%|█████████ | 91/100 [00:00<00:00, 1036.31it/s, train_auc=50.28%, skipped=10.67%]

92%|█████████▏| 92/100 [00:00<00:00, 1037.20it/s, train_auc=54.23%, skipped=10.67%]

93%|█████████▎| 93/100 [00:00<00:00, 1039.30it/s, train_auc=51.89%, skipped=11.88%]

94%|█████████▍| 94/100 [00:00<00:00, 1041.44it/s, train_auc=50.22%, skipped=9.77%]

95%|█████████▌| 95/100 [00:00<00:00, 1043.42it/s, train_auc=49.39%, skipped=9.06%]

96%|█████████▌| 96/100 [00:00<00:00, 1044.90it/s, train_auc=47.03%, skipped=11.78%]

97%|█████████▋| 97/100 [00:00<00:00, 1046.82it/s, train_auc=51.47%, skipped=10.98%]

98%|█████████▊| 98/100 [00:00<00:00, 1048.13it/s, train_auc=52.78%, skipped=12.99%]

99%|█████████▉| 99/100 [00:00<00:00, 1046.14it/s, train_auc=52.44%, skipped=11.28%]

100%|██████████| 100/100 [00:00<00:00, 1046.10it/s, train_auc=49.54%, skipped=11.58%]

100%|██████████| 100/100 [00:00<00:00, 1044.26it/s, train_auc=49.54%, skipped=11.58%]

W1116 19:23:15.990415 28585 pinned_memory_manager.cc:236] Unable to allocate pinned system memory, pinned memory pool will not be available: out of memory

I1116 19:23:15.991103 28585 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1116 19:23:15.993164 28585 server.cc:209] Failed to finalize CUDA memory manager: CNMEM_STATUS_OUT_OF_MEMORY

I1116 19:23:15.995865 28585 model_lifecycle.cc:459] loading: executor_model:1

1116 19:23:18.850815 28625 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: CUDARuntimeError: cudaErrorInitializationError: initialization error


At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/tmp/pytest-of-jenkins/pytest-21/test_ensemble_BayesianPersonal1/executor_model/1/model.py(34): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1116 19:23:18.878293 28585 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error


At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/tmp/pytest-of-jenkins/pytest-21/test_ensemble_BayesianPersonal1/executor_model/1/model.py(34): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


I1116 19:23:18.878546 28585 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1116 19:23:18.878665 28585 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 19:23:18.878904 28585 server.cc:631]

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model          | Version | Status                                                                                                                                                   |

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

| executor_model | 1       | UNAVAILABLE: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error                                                              |

|                |         |                                                                                                                                                          |

|                |         | At:                                                                                                                                                      |

|                |         |   /usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount                                                                           |

|                |         |   /usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup                                                                     |

|                |         |   /usr/local/lib/python3.8/dist-packages/cudf/init.py(5):                                                                                    |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51):                      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22):                      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18):  |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18):      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (961): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26):               |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20):               |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /tmp/pytest-of-jenkins/pytest-21/test_ensemble_BayesianPersonal1/executor_model/1/model.py(34):                                                |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 19:23:18.946229 28585 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1116 19:23:18.947076 28585 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-21/test_ensemble_BayesianPersonal1                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1116 19:23:18.947116 28585 server.cc:262] Waiting for in-flight requests to complete.

I1116 19:23:18.947123 28585 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1116 19:23:18.947129 28585 server.cc:293] All models are stopped, unloading models

I1116 19:23:18.947135 28585 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1116 19:23:20.043653 28585 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            222     31    86%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         82      5    94%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        65     19    71%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      54      5    91%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2749   1184    57%


============ 1 failed, 90 passed, 55 warnings in 1261.85s (0:21:01) ============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins12240847061300192550.sh

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #230 of commit 0492961e7d92c2321f1d13d23c5ecb4fae94476d, no merge conflicts.
Running as SYSTEM
Setting status of 0492961e7d92c2321f1d13d23c5ecb4fae94476d to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/525/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/230/*:refs/remotes/origin/pr/230/* # timeout=10
 > git rev-parse 0492961e7d92c2321f1d13d23c5ecb4fae94476d^{commit} # timeout=10
Checking out Revision 0492961e7d92c2321f1d13d23c5ecb4fae94476d (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0492961e7d92c2321f1d13d23c5ecb4fae94476d # timeout=10
Commit message: "Merge branch 'main' of github.com:AyodeAwe/systems into gpuci-jenkinsfile"
 > git rev-list --no-walk 8024664acb7da8110ccd1316b6e9fb5f752cd85c # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins15543640262559400195.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+19.g0492961.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.13,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.13,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g0492961,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@e5b7351deb9e4885c4038aa0bbc9f146d8477a0e,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1385337840'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-24q6t4rm
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-24q6t4rm
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 04ff5225cee6af90b6b08070493677ac53b96836
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+16.g04ff522) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+16.g04ff522) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+16.g04ff522) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+16.g04ff522) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+16.g04ff522) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+16.g04ff522) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+16.g04ff522) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+16.g04ff522) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+16.g04ff522) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+16.g04ff522) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+16.g04ff522) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+16.g04ff522) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+16.g04ff522) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (2.0.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+16.g04ff522) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+16.g04ff522) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+16.g04ff522) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+16.g04ff522) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+16.g04ff522) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+16.g04ff522) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+16.g04ff522) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+16.g04ff522) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+16.g04ff522) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+16.g04ff522) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+16.g04ff522) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+16.g04ff522) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+16.g04ff522) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+16.g04ff522) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+16.g04ff522-py3-none-any.whl size=118833 sha256=0d20852212422fea2514271c20fc3bf04eabb2168866cfde1a58a5fa072f17b1
  Stored in directory: /tmp/pip-ephem-wheel-cache-8wcp7k_b/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+16.g04ff522
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g0492961) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g0492961) (0.8.0+16.g04ff522)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.g0492961) (2.22.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.10.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2.0.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g0492961) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g0492961-py3-none-any.whl size=102183 sha256=eb53c54fa492dcf97b0dac47082378ea66ca098286cfead6f9e7b0cd1af6c083
  Stored in directory: /tmp/pip-ephem-wheel-cache-mp5nke0t/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+19.g0492961
    Uninstalling merlin-systems-0.7.0+19.g0492961:
      Successfully uninstalled merlin-systems-0.7.0+19.g0492961
Successfully installed merlin-systems-0.7.0+19.g0492961
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 91 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 12%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 27%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 39%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            222     31    86%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         82      5    94%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        65     19    71%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      54      5    91%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     14    81%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2750   1187    57%


================= 91 passed, 55 warnings in 1080.80s (0:18:00) =================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins8032760802351336554.sh

@AyodeAwe AyodeAwe marked this pull request as ready for review November 22, 2022 12:19
@karlhigley karlhigley added chore Maintenance for the repository ci labels Nov 22, 2022
@karlhigley karlhigley merged commit 1e2dc48 into NVIDIA-Merlin:main Nov 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@karlhigley karlhigley karlhigley approved these changes

Assignees
No one assigned
Labels
chore Maintenance for the repository ci gpuCI-test-this Enables PR to be discovered and tested on gpuci.io
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@AyodeAwe @nvidia-merlin-bot @karlhigley