Refactor TensorFlow operator into base and Triton runtime operator #219

oliverholworthy · 2022-10-27T18:20:04Z

Extract Triton parts of PredictTensorflow into a Triton-specific operator and integrate with the Triton runtime.

Co-authored-by: Karl Higley <[email protected]> Co-authored-by: Julio Perez <[email protected]>

Co-authored-by: Julio Perez <[email protected]> Co-authored-by: Karl Higley <[email protected]>

github-actions · 2022-10-27T18:29:37Z

Documentation preview

https://nvidia-merlin.github.io/systems/review/pr-219

karlhigley · 2022-10-27T19:39:40Z

merlin/systems/dag/ops/operator.py

-            ),
-            os.path.join(node_export_path, str(version), "model.py"),
-        )
+        with importlib.resources.path(


TIL 👍🏻

oliverholworthy · 2022-10-27T20:15:29Z

merlin/systems/triton/models/executor_model.py

@@ -119,7 +119,8 @@ def execute(self, requests):
                tb_string = repr(traceback.extract_tb(exc_traceback))
                responses.append(
                    pb_utils.InferenceResponse(
-                        tensors=[], error=f"{exc_type}, {exc_value}, {tb_string}"
+                        output_tensors=[],


This change corrects the arguments passed to InferenceResponse which makes the error messages a bit clearer when something goes wrong. Previously errors were all wrapped in a message about arguments to the InferenceResponse which made things more difficult to quickly scan and see what was going wrong:

Failed to process the request(s) for model instance '1_predictforest', message: TypeError: __init__(): incompatible constructor arguments. The following argument types are supported: E 1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None) E E Invoked with: kwargs: tensors=[], error="'...

Example

Before

After

oliverholworthy · 2022-10-28T08:36:42Z

rerun tests

nvidia-merlin-bot · 2022-10-28T08:52:37Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/465/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins13233484400888250928.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='642131393'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-lry5fd_b
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-lry5fd_b
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 2c621a26a2b1b7ed786c99bd7c2790b9fc675098
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (6.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (65.4.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+3.g2c621a2) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+3.g2c621a2-py3-none-any.whl size=118254 sha256=8da79f88f1d5aee8ba24f80cdf99ff65810bfea00f5e9cd4eda5cfe2007fc32a
  Stored in directory: /tmp/pip-ephem-wheel-cache-ovc2u38v/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+3.g2c621a2
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.1.1)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.g736699e) (2.22.0)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0+3.g2c621a2)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.19.5)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.55.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.4.1)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.8.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.0.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97682 sha256=feedc96c3a1300546b2b9e67de612d5144731305b5ad8e667e7c13fa7bd4e375

Stored in directory: /tmp/pip-ephem-wheel-cache-s07xipnz/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+19.g736699e

Uninstalling merlin-systems-0.7.0+19.g736699e:

Successfully uninstalled merlin-systems-0.7.0+19.g736699e

Successfully installed merlin-systems-0.7.0+19.g736699e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 92 items
tests/unit/test_export.py . tests/unit/examples/test_serving_an_implicit_m . tests/unit/examples/test_serving_an_xgboost_mo [  4%]
 tests/unit/examples/test_serving_ranking_model [  5%]
 tests/unit/systems/dag/test_ensemble.py . tests/unit/systems/dag/test_executors.py ... tests/unit/systems/dag/test_graph.py .. tests/unit/systems/dag/test_model_registry.py .. tests/unit/systems/dag/test_op_runner.py .... tests/unit/systems/dag/ops/test_ops.py .. tests/unit/systems/dag/runtimes/test_triton.py ..... tests/unit/systems/ops/faiss/test_executor.py F tests/unit/systems/ops/feast/test_op.py ....... tests/unit/systems/ops/fil/test_ensemble.py . tests/unit/systems/ops/fil/test_forest.py .... tests/unit/systems/ops/fil/test_op.py tests/unit/systems/ops/implicit/test_executor.py ... tests/unit/systems/ops/implicit/test_op.py .....F... tests/unit/systems/ops/nvtabular/test_ensemble.py . tests/unit/systems/ops/nvtabular/test_op.py .. tests/unit/systems/ops/tf/test_ensemble.py .... tests/unit/systems/ops/tf/test_op.py .... tests/unit/systems/ops/torch/test_op.py ......

[  1%]
 odel_with_merlin_systems.py . [  2%]
 [  3%]
 del_with_merlin_systems.py . [  4%]
 s_with_merlin_systems.py . [  5%]
 [  6%]
 [  9%]
 [ 11%]
 [ 14%]
 [ 18%]
 [ 20%]
 [ 26%]
 [ 27%]
 [ 34%]
 [ 35%]
 [ 40%]
 ..........................         [ 68%]
 [ 71%]
 [ 81%]
 [ 82%]
 [ 84%]
 [ 89%]
 [ 93%]
 [100%]
=================================== FAILURES ===================================

_____________________ test_faiss_in_triton_executor_model ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble_config, _ = ensemble.export(tmpdir, runtime=TritonExecutorRuntime())


  response = run_ensemble_on_tritonserver(


        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        ensemble_config.name,
    )

tests/unit/systems/ops/faiss/test_executor.py:81:

merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1028 08:44:43.770525 5656 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe776000000' with size 268435456

I1028 08:44:43.771272 5656 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 08:44:43.775300 5656 model_lifecycle.cc:459] loading: 0_predicttensorflowtriton:1

I1028 08:44:43.775342 5656 model_lifecycle.cc:459] loading: executor_model:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 08:44:51.897080 5656 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1028 08:44:51.897129 5656 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1028 08:44:51.897136 5656 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1028 08:44:51.897142 5656 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1028 08:44:51.897183 5656 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

2022-10-28 08:44:58.268882: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 08:44:59.437230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4404 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

1028 08:45:00.244334 5735 pb_stub.cc:309] Failed to initialize Python stub: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x562399f04d60 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 08:45:01.435785 5656 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 0_predicttensorflowtriton (version 1)

2022-10-28 08:45:01.436464: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 08:45:01.437343: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 08:45:01.437371: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 08:45:01.437485: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

E1028 08:45:01.445530 5656 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x562399f04d60 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
2022-10-28 08:45:01.482126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4854 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 08:45:01.521333: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-28 08:45:01.521940: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 08:45:01.566584: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 08:45:01.573822: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 137377 microseconds.

I1028 08:45:01.575993 5656 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 0_predicttensorflowtriton (GPU device 0)

2022-10-28 08:45:01.576361: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 08:45:01.577023: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 08:45:01.577043: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 08:45:01.578938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4854 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 08:45:01.580433: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 08:45:01.609505: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 08:45:01.618834: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 42475 microseconds.

I1028 08:45:01.619124 5656 model_lifecycle.cc:693] successfully loaded '0_predicttensorflowtriton' version 1

I1028 08:45:01.619261 5656 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 08:45:01.619379 5656 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 08:45:01.619577 5656 server.cc:631]

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model                     | Version | Status                                                                                                                                                                                                                                                                                                                                                                                                       |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predicttensorflowtriton | 1       | READY                                                                                                                                                                                                                                                                                                                                                                                                        |

| executor_model            | 1       | UNAVAILABLE: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x562399f04d60 size 1610612736 bytes (cudaMalloc error out of memory [2]) |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         | At:                                                                                                                                                                                                                                                                                                                                                                                                          |

|                           |         |   /usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu                                                                                                                                                                                                                                                                                                                         |

|                           |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts                                                                                                                                                                                                                                                           |

|                           |         |   /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize                                                                                                                                                                                                                                                                                                  |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 08:45:01.684443 5656 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 08:45:01.685284 5656 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 08:45:01.685321 5656 server.cc:262] Waiting for in-flight requests to complete.

I1028 08:45:01.685328 5656 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 08:45:01.685364 5656 server.cc:293] All models are stopped, unloading models

I1028 08:45:01.685372 5656 server.cc:300] Timeout 30: Found 1 live models and 0 in-flight non-inference requests

I1028 08:45:01.685481 5656 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 08:45:01.685601 5656 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1028 08:45:01.687944 5656 model_lifecycle.cc:578] successfully unloaded '0_predicttensorflowtriton' version 1

I1028 08:45:02.685454 5656 server.cc:300] Timeout 29: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 08:45:02.701288 5656 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

_________________ test_ensemble[AlternatingLeastSquares-None] __________________
model_cls = <function AlternatingLeastSquares at 0x7f1ed2382670>, runtime = None

tmpdir = local('/tmp/pytest-of-jenkins/pytest-2/test_ensemble_AlternatingLeast0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("runtime", [None, TritonExecutorRuntime()])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, runtime, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema)
    ensemble_config, _ = triton_ens.export(tmpdir, runtime=runtime)

    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None


  with run_triton_server(tmpdir) as client:


tests/unit/systems/ops/implicit/test_op.py:121:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-2/test_ensemble_AlternatingLeast0')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------
0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2606.56it/s]

I1028 08:47:54.516142 7951 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f34d6000000' with size 268435456

I1028 08:47:54.516876 7951 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 08:47:54.520267 7951 model_lifecycle.cc:459] loading: 0_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 08:48:05.089476 7951 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_predictimplicit (GPU device 0)

1028 08:48:11.948352 8030 pb_stub.cc:309] Failed to initialize Python stub: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu

/usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init

/tmp/pytest-of-jenkins/pytest-2/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(

E1028 08:48:12.967642 7951 model_lifecycle.cc:596] failed to load '0_predictimplicit' version 1: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu

/usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init

/tmp/pytest-of-jenkins/pytest-2/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize
E1028 08:48:12.967793 7951 model_repository_manager.cc:481] Invalid argument: ensemble 'ensemble_model' depends on '0_predictimplicit' which has no loaded version

I1028 08:48:12.967896 7951 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 08:48:12.967980 7951 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 08:48:12.968161 7951 server.cc:631]

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model             | Version | Status                                                                                                                                                                                                                                    |

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predictimplicit | 1       | UNAVAILABLE: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory |

|                   |         |                                                                                                                                                                                                                                           |

|                   |         | At:                                                                                                                                                                                                                                       |

|                   |         |   _cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit                                                                                                                                                                                     |

|                   |         |   /usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu                                                                                                                                                                 |

|                   |         |   /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load                                                                                                                                             |

|                   |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config                                                                                       |

|                   |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init                                                                                              |

|                   |         |   /tmp/pytest-of-jenkins/pytest-2/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize                                                                                                                            |

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 08:48:13.033399 7951 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 08:48:13.034270 7951 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-2/test_ensemble_AlternatingLeast0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 08:48:13.034284 7951 server.cc:262] Waiting for in-flight requests to complete.

I1028 08:48:13.034291 7951 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 08:48:13.034309 7951 server.cc:293] All models are stopped, unloading models

I1028 08:48:13.034315 7951 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 08:48:14.058968 7951 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     73    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1137    56%
============ 2 failed, 90 passed, 56 warnings in 895.87s (0:14:55) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins9419151331036761086.sh

oliverholworthy · 2022-10-28T12:14:26Z

rerun tests

nvidia-merlin-bot · 2022-10-28T12:31:22Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/466/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins14343867176048426866.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='658954166'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-pin0_rh7
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-pin0_rh7
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 2c621a26a2b1b7ed786c99bd7c2790b9fc675098
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.4.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+3.g2c621a2) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+3.g2c621a2-py3-none-any.whl size=118254 sha256=89a33a1bb3f7905c1f7e79f38f47a11efb4ef5cc9e7f3db3e3a0079a265e98e7
  Stored in directory: /tmp/pip-ephem-wheel-cache-ebsewic0/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+3.g2c621a2
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0+3.g2c621a2)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.g736699e) (2.22.0)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.1.1)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.55.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.19.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (7.0.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.0.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.8.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.38.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97682 sha256=85e484d449db6dec4b9778a5f792c5bba448259c0932771d0ffcac75dcb8bcc4

Stored in directory: /tmp/pip-ephem-wheel-cache-dx8qng61/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+19.g736699e

Uninstalling merlin-systems-0.7.0+19.g736699e:

Successfully uninstalled merlin-systems-0.7.0+19.g736699e

Successfully installed merlin-systems-0.7.0+19.g736699e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 92 items
tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 11%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 27%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]
=================================== FAILURES ===================================

_____________________ test_faiss_in_triton_executor_model ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble_config, _ = ensemble.export(tmpdir, runtime=TritonExecutorRuntime())


  response = run_ensemble_on_tritonserver(


        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        ensemble_config.name,
    )

tests/unit/systems/ops/faiss/test_executor.py:81:

merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1028 12:23:54.425036 4956 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f327e000000' with size 268435456

I1028 12:23:54.425751 4956 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 12:23:54.429598 4956 model_lifecycle.cc:459] loading: 0_predicttensorflowtriton:1

I1028 12:23:54.429637 4956 model_lifecycle.cc:459] loading: executor_model:1

I1028 12:23:54.710899 4956 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1028 12:23:54.710938 4956 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1028 12:23:54.710945 4956 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1028 12:23:54.710951 4956 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1028 12:23:54.710988 4956 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 0_predicttensorflowtriton (version 1)

2022-10-28 12:23:54.711532: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 12:23:54.712334: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 12:23:54.712360: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 12:23:54.712450: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 12:23:55.109339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6249 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 12:23:55.138608: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-28 12:23:55.139213: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 12:23:55.159859: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 12:23:55.166879: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 455357 microseconds.

I1028 12:23:55.172858 4956 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 0_predicttensorflowtriton (GPU device 0)

2022-10-28 12:23:55.173356: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 12:23:55.174004: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 12:23:55.174024: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 12:23:55.175960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6249 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 12:23:55.177398: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 12:23:55.198774: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 12:23:55.205940: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 32587 microseconds.

I1028 12:23:55.206182 4956 model_lifecycle.cc:693] successfully loaded '0_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 12:24:03.026034 4956 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

2022-10-28 12:24:09.013305: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 12:24:10.132175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5795 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

1028 12:24:10.948936 5073 pb_stub.cc:309] Failed to initialize Python stub: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x55ec9dfc4830 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

E1028 12:24:11.885671 4956 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x55ec9dfc4830 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
I1028 12:24:11.885925 4956 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 12:24:11.886076 4956 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 12:24:11.886288 4956 server.cc:631]

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model                     | Version | Status                                                                                                                                                                                                                                                                                                                                                                                                       |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predicttensorflowtriton | 1       | READY                                                                                                                                                                                                                                                                                                                                                                                                        |

| executor_model            | 1       | UNAVAILABLE: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x55ec9dfc4830 size 1610612736 bytes (cudaMalloc error out of memory [2]) |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         | At:                                                                                                                                                                                                                                                                                                                                                                                                          |

|                           |         |   /usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu                                                                                                                                                                                                                                                                                                                         |

|                           |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts                                                                                                                                                                                                                                                           |

|                           |         |   /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize                                                                                                                                                                                                                                                                                                  |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 12:24:11.950014 4956 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 12:24:11.950868 4956 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-0/test_faiss_in_triton_executor_0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 12:24:11.950904 4956 server.cc:262] Waiting for in-flight requests to complete.

I1028 12:24:11.950914 4956 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 12:24:11.950967 4956 server.cc:293] All models are stopped, unloading models

I1028 12:24:11.950976 4956 server.cc:300] Timeout 30: Found 1 live models and 0 in-flight non-inference requests

I1028 12:24:11.951105 4956 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 12:24:11.951451 4956 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1028 12:24:11.955770 4956 model_lifecycle.cc:578] successfully unloaded '0_predicttensorflowtriton' version 1

I1028 12:24:12.951058 4956 server.cc:300] Timeout 29: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 12:24:12.974653 4956 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     73    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1137    56%
============ 1 failed, 91 passed, 56 warnings in 869.21s (0:14:29) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins6541034653239421615.sh

karlhigley · 2022-10-28T13:51:51Z

rerun tests

nvidia-merlin-bot · 2022-10-28T14:07:37Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/467/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins10881713473283587429.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1858486568'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-4f07k7zo
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-4f07k7zo
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 2c621a26a2b1b7ed786c99bd7c2790b9fc675098
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+3.g2c621a2) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+3.g2c621a2) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (5.4.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.7.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+3.g2c621a2) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+3.g2c621a2) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+3.g2c621a2) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+3.g2c621a2) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+3.g2c621a2) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+3.g2c621a2-py3-none-any.whl size=118254 sha256=9bd5ff74db42681a288c120cca894586230aefce39439b705bbc02b01b8cd9be
  Stored in directory: /tmp/pip-ephem-wheel-cache-69iaqdle/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+3.g2c621a2
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.1.1)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.g736699e) (2.22.0)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0+3.g2c621a2)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.19.5)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.55.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.4.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.0.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.8.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.38.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97682 sha256=f4b4b377187b7eac54071e59fe15b80a79c842a0386ce32c41bfbb3cda938c7a

Stored in directory: /tmp/pip-ephem-wheel-cache-emcd81eg/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+19.g736699e

Uninstalling merlin-systems-0.7.0+19.g736699e:

Successfully uninstalled merlin-systems-0.7.0+19.g736699e

Successfully installed merlin-systems-0.7.0+19.g736699e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 92 items
tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 11%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 27%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]
=================================== FAILURES ===================================

_____________________ test_faiss_in_triton_executor_model ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble_config, _ = ensemble.export(tmpdir, runtime=TritonExecutorRuntime())


  response = run_ensemble_on_tritonserver(


        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        ensemble_config.name,
    )

tests/unit/systems/ops/faiss/test_executor.py:81:

merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1028 13:59:35.800065 18451 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f3044000000' with size 268435456

I1028 13:59:35.800829 18451 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 13:59:35.805294 18451 model_lifecycle.cc:459] loading: 0_predicttensorflowtriton:1

I1028 13:59:35.805382 18451 model_lifecycle.cc:459] loading: executor_model:1

I1028 13:59:36.072404 18451 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1028 13:59:36.072446 18451 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1028 13:59:36.072453 18451 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1028 13:59:36.072458 18451 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1028 13:59:36.072496 18451 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 0_predicttensorflowtriton (version 1)

2022-10-28 13:59:36.073032: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 13:59:36.073829: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 13:59:36.073855: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 13:59:36.073943: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 13:59:36.480488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6249 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 13:59:36.515583: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-28 13:59:36.516188: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 13:59:36.536850: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 13:59:36.543879: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 470858 microseconds.

I1028 13:59:36.549764 18451 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 0_predicttensorflowtriton (GPU device 0)

2022-10-28 13:59:36.550245: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 13:59:36.550907: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 13:59:36.550927: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 13:59:36.552846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6249 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 13:59:36.554281: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 13:59:36.575557: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 13:59:36.582755: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 32513 microseconds.

I1028 13:59:36.583016 18451 model_lifecycle.cc:693] successfully loaded '0_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 13:59:44.404479 18451 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

2022-10-28 13:59:50.366417: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 13:59:51.480490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5795 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

1028 13:59:52.291532 18572 pb_stub.cc:309] Failed to initialize Python stub: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x559f1ed12140 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

E1028 13:59:53.253528 18451 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x559f1ed12140 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
I1028 13:59:53.253721 18451 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 13:59:53.253866 18451 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 13:59:53.254080 18451 server.cc:631]

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model                     | Version | Status                                                                                                                                                                                                                                                                                                                                                                                                       |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predicttensorflowtriton | 1       | READY                                                                                                                                                                                                                                                                                                                                                                                                        |

| executor_model            | 1       | UNAVAILABLE: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x559f1ed12140 size 1610612736 bytes (cudaMalloc error out of memory [2]) |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         | At:                                                                                                                                                                                                                                                                                                                                                                                                          |

|                           |         |   /usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu                                                                                                                                                                                                                                                                                                                         |

|                           |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts                                                                                                                                                                                                                                                           |

|                           |         |   /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize                                                                                                                                                                                                                                                                                                  |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 13:59:53.319397 18451 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 13:59:53.320262 18451 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-2/test_faiss_in_triton_executor_0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 13:59:53.320299 18451 server.cc:262] Waiting for in-flight requests to complete.

I1028 13:59:53.320308 18451 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 13:59:53.320347 18451 server.cc:293] All models are stopped, unloading models

I1028 13:59:53.320355 18451 server.cc:300] Timeout 30: Found 1 live models and 0 in-flight non-inference requests

I1028 13:59:53.320530 18451 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 13:59:53.320893 18451 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1028 13:59:53.324908 18451 model_lifecycle.cc:578] successfully unloaded '0_predicttensorflowtriton' version 1

I1028 13:59:54.320439 18451 server.cc:300] Timeout 29: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 13:59:54.341639 18451 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     73    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1137    56%
============ 1 failed, 91 passed, 56 warnings in 885.20s (0:14:45) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins17048616433812767957.sh

jperez999 · 2022-10-28T14:32:48Z

rerun tests

nvidia-merlin-bot · 2022-10-28T14:49:21Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/469/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins3410026490360081085.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1641268801'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ii0u59j5
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ii0u59j5
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit eda153c663aa864da66927c7a0a9d4e64c073120
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (3.19.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.10.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (21.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (8.1.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (65.4.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+4.geda153c) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+4.geda153c-py3-none-any.whl size=118258 sha256=7400e61adc7b9f8ea6daef6e2b0283fe41933614581133d92f3eb2cf3f5ceafd
  Stored in directory: /tmp/pip-ephem-wheel-cache-jwv1bcu1/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+4.geda153c
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.1.1)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0+4.geda153c)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.g736699e) (2.22.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.55.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (7.0.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.19.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.0.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.8.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.38.1)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97682 sha256=ed0b7b8c0044efe8d5a538751ea3fd642ef85e8a51ae19437b46c809d57449d3

Stored in directory: /tmp/pip-ephem-wheel-cache-h6izk4r0/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+19.g736699e

Uninstalling merlin-systems-0.7.0+19.g736699e:

Successfully uninstalled merlin-systems-0.7.0+19.g736699e

Successfully installed merlin-systems-0.7.0+19.g736699e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 92 items
tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 11%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 27%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]
=================================== FAILURES ===================================

_____________________ test_faiss_in_triton_executor_model ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble_config, _ = ensemble.export(tmpdir, runtime=TritonExecutorRuntime())


  response = run_ensemble_on_tritonserver(


        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        ensemble_config.name,
    )

tests/unit/systems/ops/faiss/test_executor.py:81:

merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1028 14:41:11.728017 21190 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7effc6000000' with size 268435456

I1028 14:41:11.729489 21190 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 14:41:11.736206 21190 model_lifecycle.cc:459] loading: 0_predicttensorflowtriton:1

I1028 14:41:11.736280 21190 model_lifecycle.cc:459] loading: executor_model:1

I1028 14:41:12.046200 21190 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1028 14:41:12.046239 21190 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1028 14:41:12.046246 21190 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1028 14:41:12.046252 21190 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1028 14:41:12.046294 21190 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 0_predicttensorflowtriton (version 1)

2022-10-28 14:41:12.046836: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 14:41:12.047638: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 14:41:12.047664: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 14:41:12.047755: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 14:41:12.586649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6249 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 14:41:12.619135: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-28 14:41:12.619750: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 14:41:12.666499: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 14:41:12.732718: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 685892 microseconds.

I1028 14:41:12.738204 21190 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 0_predicttensorflowtriton (GPU device 0)

2022-10-28 14:41:12.738723: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 14:41:12.739366: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 14:41:12.739386: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 14:41:12.741290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6249 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 14:41:12.742700: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 14:41:12.763996: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 14:41:12.771222: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 32501 microseconds.

I1028 14:41:12.771447 21190 model_lifecycle.cc:693] successfully loaded '0_predicttensorflowtriton' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 14:41:20.644644 21190 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

2022-10-28 14:41:26.757685: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 14:41:28.436880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5795 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

1028 14:41:29.424250 21312 pb_stub.cc:309] Failed to initialize Python stub: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x5558e750f630 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

E1028 14:41:30.492563 21190 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x5558e750f630 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
I1028 14:41:30.492752 21190 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 14:41:30.492826 21190 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 14:41:30.492940 21190 server.cc:631]

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model                     | Version | Status                                                                                                                                                                                                                                                                                                                                                                                                       |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predicttensorflowtriton | 1       | READY                                                                                                                                                                                                                                                                                                                                                                                                        |

| executor_model            | 1       | UNAVAILABLE: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x5558e750f630 size 1610612736 bytes (cudaMalloc error out of memory [2]) |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         | At:                                                                                                                                                                                                                                                                                                                                                                                                          |

|                           |         |   /usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu                                                                                                                                                                                                                                                                                                                         |

|                           |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts                                                                                                                                                                                                                                                           |

|                           |         |   /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize                                                                                                                                                                                                                                                                                                  |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 14:41:30.636721 21190 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 14:41:30.637557 21190 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-1/test_faiss_in_triton_executor_0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 14:41:30.637591 21190 server.cc:262] Waiting for in-flight requests to complete.

I1028 14:41:30.637601 21190 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 14:41:30.637649 21190 server.cc:293] All models are stopped, unloading models

I1028 14:41:30.637658 21190 server.cc:300] Timeout 30: Found 1 live models and 0 in-flight non-inference requests

I1028 14:41:30.637831 21190 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 14:41:30.638191 21190 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1028 14:41:30.642211 21190 model_lifecycle.cc:578] successfully unloaded '0_predicttensorflowtriton' version 1

I1028 14:41:31.637743 21190 server.cc:300] Timeout 29: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 14:41:31.663980 21190 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     73    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1137    56%
============ 1 failed, 91 passed, 56 warnings in 930.05s (0:15:30) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins823844120306790782.sh

jperez999 · 2022-10-28T15:05:04Z

rerun tests

nvidia-merlin-bot · 2022-10-28T15:37:04Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/470/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins17837478319592186654.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
python3.8 create: /var/jenkins_home/workspace/merlin_systems/systems/.tox/python3.8
python3.8 inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
python3.8 installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular==1.4.0+8.g95e12d347,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
python3.8 run-test-pre: PYTHONHASHSEED='1708343409'
python3.8 run-test: commands[0] | pip install --upgrade pip
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./.tox/python3.8/lib/python3.8/site-packages (22.2.2)
Collecting pip
  Downloading pip-22.3-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 13.8 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.2
    Uninstalling pip-22.2.2:
      Successfully uninstalled pip-22.2.2
Successfully installed pip-22.3
python3.8 run-test: commands[1] | pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.5.0)
Requirement already satisfied: requests<3,>=2.10 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (2.28.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0)
Requirement already satisfied: pyarrow>=5.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (10.0.0)
Requirement already satisfied: numba>=0.54 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.56.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.0)
Requirement already satisfied: betterproto<2.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.20.3)
Requirement already satisfied: packaging in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.0)
Requirement already satisfied: scipy in ./.tox/python3.8/lib/python3.8/site-packages (from nvtabular>=1.0.0->merlin-systems==0.7.0+19.g736699e) (1.9.3)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2022.9.24)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (1.26.12)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2.1.1)
Requirement already satisfied: grpclib in ./.tox/python3.8/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)
Requirement already satisfied: stringcase in ./.tox/python3.8/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)
Requirement already satisfied: cloudpickle>=1.1.1 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0)
Requirement already satisfied: jinja2 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)
Requirement already satisfied: locket>=1.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.0)
Requirement already satisfied: tornado<6.2,>=6.0.3 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)
Requirement already satisfied: psutil>=5.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.9.3)
Requirement already satisfied: click>=6.6 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)
Requirement already satisfied: numpy<1.24,>=1.18 in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.23.4)
Requirement already satisfied: importlib-metadata in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.0.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.39.1)
Requirement already satisfied: setuptools in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/python3.8/lib/python3.8/site-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in ./.tox/python3.8/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5)
Requirement already satisfied: python-dateutil>=2.7.3 in ./.tox/python3.8/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/python3.8/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.56.4)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/python3.8/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)
Requirement already satisfied: six>=1.5 in ./.tox/python3.8/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.16.0)
Requirement already satisfied: heapdict in ./.tox/python3.8/lib/python3.8/site-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)
Requirement already satisfied: multidict in ./.tox/python3.8/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/python3.8/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)
Requirement already satisfied: zipp>=0.5 in ./.tox/python3.8/lib/python3.8/site-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.10.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.tox/python3.8/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97669 sha256=5791d47fabafde30af9c92e8e92f0867e5ba663a0303b15db97e5e3b3d3118e1
  Stored in directory: /tmp/pip-ephem-wheel-cache-ndzoet18/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+19.g736699e
    Uninstalling merlin-systems-0.7.0+19.g736699e:
      Successfully uninstalled merlin-systems-0.7.0+19.g736699e
Successfully installed merlin-systems-0.7.0+19.g736699e
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1708343409'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-sfg1d9eg
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-sfg1d9eg
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit eda153c663aa864da66927c7a0a9d4e64c073120
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (7.0.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (8.1.3)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (65.4.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+4.geda153c) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+4.geda153c-py3-none-any.whl size=118258 sha256=23476c11b37ec960dc522c87806dff692a873585e8b392dfd1b9df797dd7075c
  Stored in directory: /tmp/pip-ephem-wheel-cache-u6nlh6in/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+4.geda153c
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.1.1)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+19.g736699e) (2.22.0)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0+4.geda153c)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (7.0.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.55.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.19.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.3.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.4.1)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.0.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.8.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97682 sha256=faa2b3b209231f3dfc7e9239f687c091885d2e9f245bd6da95a6f237f3f41d70

Stored in directory: /tmp/pip-ephem-wheel-cache-fz30z98d/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+19.g736699e

Uninstalling merlin-systems-0.7.0+19.g736699e:

Successfully uninstalled merlin-systems-0.7.0+19.g736699e

Successfully installed merlin-systems-0.7.0+19.g736699e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 92 items
tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 11%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 27%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]
=================================== FAILURES ===================================

_____________________ test_faiss_in_triton_executor_model ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble_config, _ = ensemble.export(tmpdir, runtime=TritonExecutorRuntime())


  response = run_ensemble_on_tritonserver(


        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        ensemble_config.name,
    )

tests/unit/systems/ops/faiss/test_executor.py:81:

merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1028 15:14:46.463811 12896 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fdf3e000000' with size 268435456

I1028 15:14:46.465133 12896 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 15:14:46.470777 12896 model_lifecycle.cc:459] loading: 0_predicttensorflowtriton:1

I1028 15:14:46.470820 12896 model_lifecycle.cc:459] loading: executor_model:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 15:14:54.590808 12896 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1028 15:14:54.590850 12896 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1028 15:14:54.590857 12896 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1028 15:14:54.590863 12896 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1028 15:14:54.590908 12896 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

2022-10-28 15:15:01.108585: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 15:15:02.681356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5689 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

1028 15:15:03.638078 12975 pb_stub.cc:309] Failed to initialize Python stub: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x55b13b57dfc0 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1028 15:15:04.633029 12896 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 0_predicttensorflowtriton (version 1)

2022-10-28 15:15:04.634411: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:15:04.636464: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 15:15:04.636524: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:15:04.636734: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

E1028 15:15:04.642416 12896 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x55b13b57dfc0 size 1610612736 bytes (cudaMalloc error out of memory [2])
At:

/usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts

/tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize
2022-10-28 15:15:04.678775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 15:15:04.714131: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-28 15:15:04.714734: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 15:15:04.755323: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:15:04.763307: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 128932 microseconds.

I1028 15:15:04.777589 12896 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 0_predicttensorflowtriton (GPU device 0)

2022-10-28 15:15:04.779603: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:15:04.780538: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 15:15:04.780567: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:15:04.857644: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6139 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 15:15:04.860181: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 15:15:04.899107: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/0_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:15:04.909007: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 129409 microseconds.

I1028 15:15:04.909267 12896 model_lifecycle.cc:693] successfully loaded '0_predicttensorflowtriton' version 1

I1028 15:15:04.909410 12896 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 15:15:04.909546 12896 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:15:04.909760 12896 server.cc:631]

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model                     | Version | Status                                                                                                                                                                                                                                                                                                                                                                                                       |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predicttensorflowtriton | 1       | READY                                                                                                                                                                                                                                                                                                                                                                                                        |

| executor_model            | 1       | UNAVAILABLE: Internal: RuntimeError: Error in virtual void* faiss::gpu::StandardGpuResourcesImpl::allocMemory(const faiss::gpu::AllocRequest&) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:452: Error: 'err == cudaSuccess' failed: StandardGpuResources: alloc fail type TemporaryMemoryBuffer dev 0 space Device stream 0x55b13b57dfc0 size 1610612736 bytes (cudaMalloc error out of memory [2]) |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         |                                                                                                                                                                                                                                                                                                                                                                                                              |

|                           |         | At:                                                                                                                                                                                                                                                                                                                                                                                                          |

|                           |         |   /usr/local/lib/python3.8/dist-packages/faiss/swigfaiss.py(10275): index_cpu_to_gpu                                                                                                                                                                                                                                                                                                                         |

|                           |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/faiss.py(74): load_artifacts                                                                                                                                                                                                                                                           |

|                           |         |   /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0/executor_model/1/model.py(71): initialize                                                                                                                                                                                                                                                                                                  |

+---------------------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:15:04.977087 12896 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 15:15:04.977941 12896 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-3/test_faiss_in_triton_executor_0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:15:04.977977 12896 server.cc:262] Waiting for in-flight requests to complete.

I1028 15:15:04.977985 12896 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 15:15:04.978027 12896 server.cc:293] All models are stopped, unloading models

I1028 15:15:04.978037 12896 server.cc:300] Timeout 30: Found 1 live models and 0 in-flight non-inference requests

I1028 15:15:04.978118 12896 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 15:15:04.978205 12896 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1028 15:15:04.979966 12896 model_lifecycle.cc:578] successfully unloaded '0_predicttensorflowtriton' version 1

I1028 15:15:05.978119 12896 server.cc:300] Timeout 29: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 15:15:05.997748 12896 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     73    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1137    56%
============ 1 failed, 91 passed, 56 warnings in 918.13s (0:15:18) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

test-cpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu

test-cpu installdeps: -rrequirements/test-cpu.txt

WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.

test-cpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+19.g736699e.zip

WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.

test-cpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.3,Babel==2.10.3,backcall==0.2.0,bandit==1.7.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.3,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,codespell==2.2.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cpplint==1.6.1,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,exceptiongroup==1.0.0,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-cpu==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flake8==3.9.2,flake8-nb==0.3.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,interrogate==1.5.0,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,isort==5.9.3,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mccabe==0.6.1,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular==1.4.0+8.g95e12d347,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathspec==0.10.1,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.4.1,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycodestyle==2.7.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,pyflakes==2.3.1,Pygments==2.13.0,pylint==2.7.4,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.0,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0

test-cpu run-test-pre: PYTHONHASHSEED='1708343409'

test-cpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Collecting git+https://github.com/NVIDIA-Merlin/core.git

Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ql_679ks

Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ql_679ks

Resolved https://github.com/NVIDIA-Merlin/core.git to commit eda153c663aa864da66927c7a0a9d4e64c073120

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: pyarrow>=5.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (10.0.0)

Requirement already satisfied: numba>=0.54 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (0.56.3)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.3.5)

Collecting dask>=2022.3.0

Downloading dask-2022.10.0-py3-none-any.whl (1.1 MB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 9.1 MB/s eta 0:00:00

Requirement already satisfied: betterproto<2.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.2.5)

Requirement already satisfied: tqdm>=4.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (4.64.1)

Collecting fsspec==2022.5.0

Downloading fsspec-2022.5.0-py3-none-any.whl (140 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 129.4 MB/s eta 0:00:00

Requirement already satisfied: tensorflow-metadata>=1.2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.10.0)

Requirement already satisfied: protobuf>=3.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (3.19.6)

Requirement already satisfied: packaging in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (21.3)

Collecting distributed>=2022.3.0

Downloading distributed-2022.10.0-py3-none-any.whl (908 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 908.6/908.6 kB 99.2 MB/s eta 0:00:00

Requirement already satisfied: grpclib in ./.tox/test-cpu/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (0.4.3)

Requirement already satisfied: stringcase in ./.tox/test-cpu/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.3.0)

Requirement already satisfied: cloudpickle>=1.1.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (6.0)

Requirement already satisfied: jinja2 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (3.1.2)

Requirement already satisfied: locket>=1.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.0)

Collecting tornado<6.2,>=6.0.3

Downloading tornado-6.1-cp38-cp38-manylinux2010_x86_64.whl (427 kB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.5/427.5 kB 127.5 MB/s eta 0:00:00

Requirement already satisfied: msgpack>=0.6.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.7.0)

Requirement already satisfied: zict>=0.1.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.2.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.4.0)

Requirement already satisfied: psutil>=5.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.9.3)

Requirement already satisfied: urllib3 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.26.12)

Requirement already satisfied: click>=6.6 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (8.0.4)

Requirement already satisfied: numpy<1.24,>=1.18 in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (1.23.4)

Requirement already satisfied: importlib-metadata in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (5.0.0)

Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (0.39.1)

Requirement already satisfied: setuptools in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/test-cpu/lib/python3.8/site-packages (from packaging->merlin-core==0.8.0+4.geda153c) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2022.5)

Requirement already satisfied: python-dateutil>=2.7.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/test-cpu/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.3.0)

Requirement already satisfied: six>=1.5 in ./.tox/test-cpu/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.16.0)

Requirement already satisfied: heapdict in ./.tox/test-cpu/lib/python3.8/site-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.1)

Requirement already satisfied: multidict in ./.tox/test-cpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.1.0)

Requirement already satisfied: zipp>=0.5 in ./.tox/test-cpu/lib/python3.8/site-packages (from importlib-metadata->numba>=0.54->merlin-core==0.8.0+4.geda153c) (3.10.0)

Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.0.0)

Building wheels for collected packages: merlin-core

Building wheel for merlin-core (pyproject.toml): started

Building wheel for merlin-core (pyproject.toml): finished with status 'done'

Created wheel for merlin-core: filename=merlin_core-0.8.0+4.geda153c-py3-none-any.whl size=118243 sha256=bef305f6a4775d84da8e262c45c2588465d64af1a4ec51fa3316e97e8fe38320

Stored in directory: /tmp/pip-ephem-wheel-cache-wm75ltji/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84

Successfully built merlin-core

Installing collected packages: tornado, fsspec, dask, distributed, merlin-core

Attempting uninstall: tornado

Found existing installation: tornado 6.2

Uninstalling tornado-6.2:

Successfully uninstalled tornado-6.2

Attempting uninstall: fsspec

Found existing installation: fsspec 2022.10.0

Uninstalling fsspec-2022.10.0:

Successfully uninstalled fsspec-2022.10.0

Attempting uninstall: dask

Found existing installation: dask 2022.1.1

Uninstalling dask-2022.1.1:

Successfully uninstalled dask-2022.1.1

Attempting uninstall: distributed

Found existing installation: distributed 2022.1.1

Uninstalling distributed-2022.1.1:

Successfully uninstalled distributed-2022.1.1

Attempting uninstall: merlin-core

Found existing installation: merlin-core 0.5.0

Uninstalling merlin-core-0.5.0:

Successfully uninstalled merlin-core-0.5.0

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.

jupyter-client 7.4.4 requires tornado>=6.2, but you have tornado 6.1 which is incompatible.

feast 0.19.4 requires dask<2022.02.0,>=2021.*, but you have dask 2022.10.0 which is incompatible.

Successfully installed dask-2022.10.0 distributed-2022.10.0 fsspec-2022.5.0 merlin-core-0.8.0+4.geda153c tornado-6.1
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-cpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.5.0)

Requirement already satisfied: requests<3,>=2.10 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (2.28.1)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0+4.geda153c)

Requirement already satisfied: pyarrow>=5.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (10.0.0)

Requirement already satisfied: numba>=0.54 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.56.3)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)

Requirement already satisfied: dask>=2022.3.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.0)

Requirement already satisfied: betterproto<2.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)

Requirement already satisfied: tqdm>=4.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)

Requirement already satisfied: fsspec==2022.5.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)

Requirement already satisfied: protobuf>=3.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.19.6)

Requirement already satisfied: packaging in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)

Requirement already satisfied: distributed>=2022.3.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.0)

Requirement already satisfied: scipy in ./.tox/test-cpu/lib/python3.8/site-packages (from nvtabular>=1.0.0->merlin-systems==0.7.0+19.g736699e) (1.9.3)

Requirement already satisfied: certifi>=2017.4.17 in ./.tox/test-cpu/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2022.9.24)

Requirement already satisfied: idna<4,>=2.5 in ./.tox/test-cpu/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (3.4)

Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (1.26.12)

Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/test-cpu/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: grpclib in ./.tox/test-cpu/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)

Requirement already satisfied: stringcase in ./.tox/test-cpu/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)

Requirement already satisfied: partd>=0.3.10 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)

Requirement already satisfied: cloudpickle>=1.1.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0)

Requirement already satisfied: jinja2 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)

Requirement already satisfied: locket>=1.0.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.0)

Requirement already satisfied: tornado<6.2,>=6.0.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)

Requirement already satisfied: zict>=0.1.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)

Requirement already satisfied: psutil>=5.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.9.3)

Requirement already satisfied: click>=6.6 in ./.tox/test-cpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.0.4)

Requirement already satisfied: numpy<1.24,>=1.18 in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.23.4)

Requirement already satisfied: importlib-metadata in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.0.0)

Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.39.1)

Requirement already satisfied: setuptools in ./.tox/test-cpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/test-cpu/lib/python3.8/site-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5)

Requirement already satisfied: python-dateutil>=2.7.3 in ./.tox/test-cpu/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/test-cpu/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)

Requirement already satisfied: six>=1.5 in ./.tox/test-cpu/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.16.0)

Requirement already satisfied: heapdict in ./.tox/test-cpu/lib/python3.8/site-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)

Requirement already satisfied: multidict in ./.tox/test-cpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)

Requirement already satisfied: zipp>=0.5 in ./.tox/test-cpu/lib/python3.8/site-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.10.0)

Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in ./.tox/test-cpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97669 sha256=5a84514f6c0146c54a69926aca1be41d003f46e1e8b63b18a6e4eedb990bd010

Stored in directory: /tmp/pip-ephem-wheel-cache-r0arym4h/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+19.g736699e

Uninstalling merlin-systems-0.7.0+19.g736699e:

Successfully uninstalled merlin-systems-0.7.0+19.g736699e

Successfully installed merlin-systems-0.7.0+19.g736699e
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-cpu run-test: commands[2] | python -m pytest --cov-report term --cov=merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.2.0, pluggy-1.0.0

cachedir: .tox/test-cpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.6.2, cov-4.0.0

collected 90 items / 1 skipped
tests/unit/test_export.py F                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py F [  2%]

[  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/systems/dag/test_ensemble.py .                                [  4%]

tests/unit/systems/dag/test_executors.py .ss                             [  7%]

tests/unit/systems/dag/test_graph.py ..                                  [ 10%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 12%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 16%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 18%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 24%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 25%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 33%]

tests/unit/systems/ops/fil/test_ensemble.py F                            [ 34%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 67%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py ......F..                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py F...                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]
=================================== FAILURES ===================================

_____________ test_export_run_ensemble_triton[tensorflow-parquet] ______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_export_run_ensemble_trito0')

engine = 'parquet', output_model = 'tensorflow'

df =      name-cat name-string    id  label         x         y

0       Edith       Wendy  1025    985  0.987753 -0.864615

...in   931    992 -0.661652  0.559267

1439    Wendy       Quinn  1023   1043 -0.827986  0.372057
[4320 rows x 6 columns]
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
@pytest.mark.parametrize("output_model", ["tensorflow"])
def test_export_run_ensemble_triton(tmpdir, engine, output_model, df):
    conts = ["x", "y", "id"] >> ops.FillMissing() >> ops.Normalize()
    cats = ["name-cat", "name-string"] >> ops.Categorify(cat_cache="host")
    workflow = Workflow(conts + cats)
    dataset = Dataset(df)
    workflow.fit(dataset)

    embed_shapes = get_embedding_sizes(workflow)
    cat_cols = list(embed_shapes.keys())

    if output_model == "tensorflow":
        tf_model = create_tf_model(cat_cols, [], embed_shapes)
        export_tensorflow_ensemble(tf_model, workflow, "test_name", tmpdir, [])
    elif output_model == "pytorch":
        torch_model = create_pytorch_model(cat_cols, [], embed_shapes)
        export_pytorch_ensemble(
            torch_model,
            workflow,
            {},
            "test_name",
            tmpdir,
            [],
        )

    # assert os.path.exists(os.path.join(repo, "config.pbtxt"))
    tri_df = df.iloc[:10]
    tri_df = tri_df[["x", "y", "id", "name-cat", "name-string"]]


  response = run_ensemble_on_tritonserver(


        str(tmpdir), workflow.input_schema, tri_df, ["output"], "test_name"
    )

tests/unit/test_export.py:73:

merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = '/tmp/pytest-of-jenkins/pytest-5/test_export_run_ensemble_trito0'
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():


              raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")


E                   RuntimeError: Another tritonserver is already running on localhost:8001
merlin/systems/triton/utils.py:58: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-10-28 15:27:08.030510: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 15:27:10.880802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4969 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 15:27:10.881533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14225 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-10-28 15:27:10.882130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14471 MB memory:  -> device: 2, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0e:00.0, compute capability: 6.0

2022-10-28 15:27:10.882709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14147 MB memory:  -> device: 3, name: Tesla P100-DGXS-16GB, pci bus id: 0000:0f:00.0, compute capability: 6.0

------------------------------ Captured log call -------------------------------

WARNING  absl:signature_serialization.py:146 Function _wrapped_model contains input name(s) name-cat, name-string with unsupported characters which will be renamed to name_cat, name_string in the SavedModel.

WARNING  absl:save.py:133 <nvtabular.framework_utils.tensorflow.layers.embedding.DenseFeatures object at 0x7f776140a1c0> has the same name 'DenseFeatures' as a built-in Keras object. Consider renaming <class 'nvtabular.framework_utils.tensorflow.layers.embedding.DenseFeatures'> to avoid naming conflicts when loading with tf.keras.models.load_model. If renaming is not possible, pass the object in the custom_objects parameter of the load function.

_____________________ test_example_serving_implicit[False] _____________________
self = <testbook.client.TestbookNotebookClient object at 0x7f775279edf0>

cell = {'cell_type': 'markdown', 'id': '65b7e4e8', 'metadata': {}, 'source': '## Retrieving Recommendations from Triton Infer...nse, to show the full experience.\n\nWe begin by obtaining 10 examples from our train data to include in the request.'}

kwargs = {}, cell_indexes = [18, 19, 20, 21, 22, 23, ...]

executed_cells = [{'cell_type': 'markdown', 'id': '65b7e4e8', 'metadata': {}, 'source': '## Retrieving Recommendations from Triton Infe...se, to show the full experience.\n\nWe begin by obtaining 10 examples from our train data to include in the request.'}]

idx = 19
def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:


          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)


.tox/test-cpu/lib/python3.8/site-packages/testbook/client.py:133:

args = (<testbook.client.TestbookNotebookClient object at 0x7f775279edf0>, {'cell_type': 'code', 'execution_count': 10, 'id':...s = train.compute()['user_id'].unique().sample(10).sort_values().to_frame().reset_index(drop=True)\nten_examples"}, 19)

kwargs = {}
def wrapped(*args, **kwargs):


  return just_run(coro(*args, **kwargs))


.tox/test-cpu/lib/python3.8/site-packages/nbclient/util.py:85:

coro = <coroutine object NotebookClient.async_execute_cell at 0x7f77527abdc0>
def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    try:
        loop = asyncio.get_running_loop()
    except RuntimeError:
        loop = None
    if loop is None:
        had_running_loop = False
        loop = asyncio.new_event_loop()
        asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()


  return loop.run_until_complete(coro)


.tox/test-cpu/lib/python3.8/site-packages/nbclient/util.py:60:

self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-51' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/workspace/m...'numpy.ndarray' object has no attribute 'sample'\nAttributeError: 'numpy.ndarray' object has no attribute 'sample'\n")>
def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')


  return future.result()


/usr/lib/python3.8/asyncio/base_events.py:616:

self = <testbook.client.TestbookNotebookClient object at 0x7f775279edf0>

cell = {'cell_type': 'code', 'execution_count': 10, 'id': '2d61751b', 'metadata': {'execution': {'iopub.status.busy': '2022-1...amples = train.compute()['user_id'].unique().sample(10).sort_values().to_frame().reset_index(drop=True)\nten_examples"}

cell_index = 19, execution_count = None, store_history = True
async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count
    await run_hook(
        self.on_cell_executed, cell=cell, cell_index=cell_index, execute_reply=exec_reply
    )


  await self._check_raise_for_error(cell, cell_index, exec_reply)


.tox/test-cpu/lib/python3.8/site-packages/nbclient/client.py:1019:

self = <testbook.client.TestbookNotebookClient object at 0x7f775279edf0>

cell = {'cell_type': 'code', 'execution_count': 10, 'id': '2d61751b', 'metadata': {'execution': {'iopub.status.busy': '2022-1...amples = train.compute()['user_id'].unique().sample(10).sort_values().to_frame().reset_index(drop=True)\nten_examples"}

cell_index = 19

exec_reply = {'buffers': [], 'content': {'ename': 'AttributeError', 'engine_info': {'engine_id': -1, 'engine_uuid': '6ac349b5-e786-...e, 'engine': '6ac349b5-e786-435e-a07e-7a117e43d1ee', 'started': '2022-10-28T15:27:48.317748Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(
        self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
    )
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           ten_examples = train.compute()['user_id'].unique().sample(10).sort_values().to_frame().reset_index(drop=True)

E           ten_examples

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mAttributeError�[0m                            Traceback (most recent call last)

E           Cell �[0;32mIn [10], line 1�[0m

E           �[0;32m----> 1�[0m ten_examples �[38;5;241m=�[39m �[43mtrain�[49m�[38;5;241;43m.�[39;49m�[43mcompute�[49m�[43m(�[49m�[43m)�[49m�[43m[�[49m�[38;5;124;43m'�[39;49m�[38;5;124;43muser_id�[39;49m�[38;5;124;43m'�[39;49m�[43m]�[49m�[38;5;241;43m.�[39;49m�[43munique�[49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43msample�[49m(�[38;5;241m10�[39m)�[38;5;241m.�[39msort_values()�[38;5;241m.�[39mto_frame()�[38;5;241m.�[39mreset_index(drop�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E           �[1;32m      2�[0m ten_examples

E

E           �[0;31mAttributeError�[0m: 'numpy.ndarray' object has no attribute 'sample'

E           AttributeError: 'numpy.ndarray' object has no attribute 'sample'
.tox/test-cpu/lib/python3.8/site-packages/nbclient/client.py:913: CellExecutionError
During handling of the above exception, another exception occurred:
gpu = False

tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_example_serving_implicit_0')
@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
def test_example_serving_implicit(gpu, tmpdir):
    with testbook(
        REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb",
        execute=False,
        timeout=180,
    ) as tb:
        tb.inject(
            f"""
            import os
            os.environ["OUTPUT_DATA_DIR"] = "{tmpdir}/ensemble"
            os.environ["USE_GPU"] = "{int(gpu)}"
            from unittest.mock import patch
            from merlin.datasets.synthetic import generate_data
            mock_train, mock_valid = generate_data(
                input="movielens-100k",
                num_rows=1000,
                set_sizes=(0.8, 0.2)
            )
            p1 = patch(
                "merlin.datasets.entertainment.get_movielens",
                return_value=[mock_train, mock_valid]
            )
            p1.start()
            """,
            pop=True,
        )

        tb.execute_cell(list(range(0, 18)))

        with run_triton_server(f"{tmpdir}/ensemble", grpc_port=8001):


          tb.execute_cell(list(range(18, len(tb.cells) - 2)))


tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:56:

self = <testbook.client.TestbookNotebookClient object at 0x7f775279edf0>

cell = {'cell_type': 'markdown', 'id': '65b7e4e8', 'metadata': {}, 'source': '## Retrieving Recommendations from Triton Infer...nse, to show the full experience.\n\nWe begin by obtaining 10 examples from our train data to include in the request.'}

kwargs = {}, cell_indexes = [18, 19, 20, 21, 22, 23, ...]

executed_cells = [{'cell_type': 'markdown', 'id': '65b7e4e8', 'metadata': {}, 'source': '## Retrieving Recommendations from Triton Infe...se, to show the full experience.\n\nWe begin by obtaining 10 examples from our train data to include in the request.'}]

idx = 19
def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:


          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))


E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E               ten_examples = train.compute()['user_id'].unique().sample(10).sort_values().to_frame().reset_index(drop=True)

E               ten_examples

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mAttributeError�[0m                            Traceback (most recent call last)

E               Cell �[0;32mIn [10], line 1�[0m

E               �[0;32m----> 1�[0m ten_examples �[38;5;241m=�[39m �[43mtrain�[49m�[38;5;241;43m.�[39;49m�[43mcompute�[49m�[43m(�[49m�[43m)�[49m�[43m[�[49m�[38;5;124;43m'�[39;49m�[38;5;124;43muser_id�[39;49m�[38;5;124;43m'�[39;49m�[43m]�[49m�[38;5;241;43m.�[39;49m�[43munique�[49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43msample�[49m(�[38;5;241m10�[39m)�[38;5;241m.�[39msort_values()�[38;5;241m.�[39mto_frame()�[38;5;241m.�[39mreset_index(drop�[38;5;241m=�[39m�[38;5;28;01mTrue�[39;00m)

E               �[1;32m      2�[0m ten_examples

E

E               �[0;31mAttributeError�[0m: 'numpy.ndarray' object has no attribute 'sample'

E               AttributeError: 'numpy.ndarray' object has no attribute 'sample'
.tox/test-cpu/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------

I1028 15:27:20.525799 19868 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fbf44000000' with size 268435456

I1028 15:27:20.527169 19868 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 15:27:20.534390 19868 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1028 15:27:20.534469 19868 model_lifecycle.cc:459] loading: 1_predictimplicit:1

I1028 15:27:30.995675 19868 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 1_predictimplicit (GPU device 0)

I1028 15:27:35.869200 19868 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1028 15:27:35.869473 19868 model_lifecycle.cc:693] successfully loaded '1_predictimplicit' version 1

I1028 15:27:37.771220 19868 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1028 15:27:37.771485 19868 model_lifecycle.cc:459] loading: ensemble_model:1

I1028 15:27:37.771805 19868 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1028 15:27:37.771897 19868 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 15:27:37.771944 19868 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:27:37.771985 19868 server.cc:631]

+---------------------+---------+--------+

| Model               | Version | Status |

+---------------------+---------+--------+

| 0_transformworkflow | 1       | READY  |

| 1_predictimplicit   | 1       | READY  |

| ensemble_model      | 1       | READY  |

+---------------------+---------+--------+
I1028 15:27:37.836201 19868 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 15:27:37.837089 19868 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_example_serving_implicit_0/ensemble                                                                                                                     |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:27:37.838530 19868 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1028 15:27:37.838743 19868 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1028 15:27:37.879735 19868 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1028 15:27:38.858259 19868 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:27:39.858479 19868 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:27:40.877070 19868 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1028 15:27:49.427025 19868 server.cc:262] Waiting for in-flight requests to complete.

I1028 15:27:49.427050 19868 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 15:27:49.427223 19868 server.cc:293] All models are stopped, unloading models

I1028 15:27:49.427241 19868 server.cc:300] Timeout 30: Found 3 live models and 0 in-flight non-inference requests

I1028 15:27:49.427252 19868 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1028 15:27:50.427331 19868 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

I1028 15:27:50.979263 19868 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1028 15:27:51.380400 19868 model_lifecycle.cc:578] successfully unloaded '1_predictimplicit' version 1

I1028 15:27:51.427454 19868 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/absl/logging/init.py", line 939, in close

self.stream.close()

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/ipykernel/iostream.py", line 446, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

_____________________ test_workflow_with_forest_inference ______________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_workflow_with_forest_infe0')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_workflow_with_forest_inference(tmpdir):
    rows = 200
    num_features = 16
    X, y = sklearn.datasets.make_regression(
        n_samples=rows,
        n_features=num_features,
        n_informative=num_features // 3,
        random_state=0,
    )
    feature_names = [str(i) for i in range(num_features)]
    df = pd.DataFrame(X, columns=feature_names, dtype=np.float32)
    dataset = Dataset(df)

    # Fit GBDT Model
    model = xgboost.XGBRegressor()
    model.fit(X, y)

    input_column_schemas = [ColumnSchema(col, dtype=np.float32) for col in feature_names]
    input_schema = Schema(input_column_schemas)
    selector = ColumnSelector(feature_names)

    workflow_ops = feature_names >> wf_ops.LogOp()
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    triton_chain = selector >> TransformWorkflow(workflow) >> PredictForest(model, input_schema)

    triton_ens = Ensemble(triton_chain, input_schema)

    request_df = df[:5]
    ensemble_config, _ = triton_ens.export(tmpdir)


  response = run_ensemble_on_tritonserver(


        str(tmpdir), input_schema, request_df, ["output__0"], ensemble_config.name
    )

tests/unit/systems/ops/fil/test_ensemble.py:77:

merlin/systems/triton/utils.py:139: in run_ensemble_on_tritonserver

response = send_triton_request(

merlin/systems/triton/utils.py:192: in send_triton_request

response = client.infer(triton_model, inputs, request_id=request_id, outputs=outputs)

.tox/test-cpu/lib/python3.8/site-packages/tritonclient/grpc/init.py:1431: in infer

raise_error_grpc(rpc_error)

rpc_error = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "in ensemble 'ensemble_model'...f-jenkins/pytest-5/test_workflow_with_forest_infe0/0_transformworkflow/1/model.py(126): execute\n","grpc_status":13}"


def raise_error_grpc(rpc_error):


  raise get_error_grpc(rpc_error) from None


E       tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '0_transformworkflow', message: AttributeError: 'NoneType' object has no attribute 'DataFrame'

E

E       At:

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(109): _array_to_cudf

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(74): convert_format

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(192): _transform_tensors

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(107): run_workflow

E         /tmp/pytest-of-jenkins/pytest-5/test_workflow_with_forest_infe0/0_transformworkflow/1/model.py(126): execute
.tox/test-cpu/lib/python3.8/site-packages/tritonclient/grpc/init.py:62: InferenceServerException

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------

I1028 15:31:21.608371 22676 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f60be000000' with size 268435456

I1028 15:31:21.609138 22676 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 15:31:21.614389 22676 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1028 15:31:21.614461 22676 model_lifecycle.cc:459] loading: 1_fil:1

I1028 15:31:21.614527 22676 model_lifecycle.cc:459] loading: 1_predictforest:1

I1028 15:31:21.629901 22676 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1028 15:31:21.629928 22676 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1028 15:31:21.629934 22676 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1028 15:31:21.630546 22676 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_fil (version 1)

I1028 15:31:21.635290 22676 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_fil_0 (GPU device 0)

I1028 15:31:21.689181 22676 model_lifecycle.cc:693] successfully loaded '1_fil' version 1

I1028 15:31:31.520561 22676 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1028 15:31:33.386162 22676 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 1_predictforest (GPU device 0)

I1028 15:31:33.386464 22676 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1028 15:31:38.155721 22676 model_lifecycle.cc:693] successfully loaded '1_predictforest' version 1

I1028 15:31:38.156500 22676 model_lifecycle.cc:459] loading: ensemble_model:1

I1028 15:31:38.157015 22676 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1028 15:31:38.157124 22676 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 15:31:38.157210 22676 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| fil     | /opt/tritonserver/backends/fil/libtriton_fil.so       | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:31:38.157293 22676 server.cc:631]

+---------------------+---------+--------+

| Model               | Version | Status |

+---------------------+---------+--------+

| 0_transformworkflow | 1       | READY  |

| 1_fil               | 1       | READY  |

| 1_predictforest     | 1       | READY  |

| ensemble_model      | 1       | READY  |

+---------------------+---------+--------+
I1028 15:31:38.273480 22676 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 15:31:38.274347 22676 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_workflow_with_forest_infe0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:31:38.275848 22676 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1028 15:31:38.276328 22676 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1028 15:31:38.317543 22676 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1028 15:31:39.299284 22676 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:31:40.299474 22676 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:31:41.320145 22676 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

Failed to transform operator <nvtabular.ops.logop.LogOp object at 0x7f16d15f5430>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py", line 183, in _transform_tensors

tensors, kind = convert_format(tensors, kind, workflow_node.inference_supports)

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py", line 74, in convert_format

return _array_to_cudf(tensors), Supports.GPU_DATAFRAME

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py", line 109, in _array_to_cudf

output = cudf.DataFrame()

AttributeError: 'NoneType' object has no attribute 'DataFrame'

1028 15:31:46.393535 22807 pb_stub.cc:777] Failed to process the request(s) for model '0_transformworkflow', message: AttributeError: 'NoneType' object has no attribute 'DataFrame'
At:

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(109): _array_to_cudf

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(74): convert_format

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(192): _transform_tensors

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(107): run_workflow

/tmp/pytest-of-jenkins/pytest-5/test_workflow_with_forest_infe0/0_transformworkflow/1/model.py(126): execute
I1028 15:31:46.394814 22676 server.cc:262] Waiting for in-flight requests to complete.

I1028 15:31:46.394859 22676 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 15:31:46.395171 22676 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1028 15:31:46.395244 22676 server.cc:293] All models are stopped, unloading models

I1028 15:31:46.395290 22676 server.cc:300] Timeout 30: Found 3 live models and 0 in-flight non-inference requests

I1028 15:31:46.395368 22676 instance_finalize.hpp:36] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 15:31:46.395910 22676 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state

I1028 15:31:46.395974 22676 model_lifecycle.cc:578] successfully unloaded '1_fil' version 1

I1028 15:31:47.395403 22676 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

I1028 15:31:47.819559 22676 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1028 15:31:48.299039 22676 model_lifecycle.cc:578] successfully unloaded '1_predictforest' version 1

I1028 15:31:48.395537 22676 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

_______________ test_ensemble[AlternatingLeastSquares-runtime1] ________________
model_cls = <function AlternatingLeastSquares at 0x7f77bfd358b0>

runtime = <merlin.systems.dag.runtimes.triton.runtime.TritonExecutorRuntime object at 0x7f7761530880>

tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1')
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("runtime", [None, TritonExecutorRuntime()])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, runtime, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema)
    ensemble_config, _ = triton_ens.export(tmpdir, runtime=runtime)

    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None


  with run_triton_server(tmpdir) as client:


tests/unit/systems/ops/implicit/test_op.py:121:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)

model_repository = local('/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1')
@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode


                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")


E                               RuntimeError: Tritonserver failed to start (ret=1)
merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------
0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2134.94it/s]

I1028 15:33:57.676583 23778 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f23a6000000' with size 268435456

I1028 15:33:57.677388 23778 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 15:33:57.680817 23778 model_lifecycle.cc:459] loading: executor_model:1

I1028 15:34:04.153434 23778 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

1028 15:34:09.318598 23825 pb_stub.cc:309] Failed to initialize Python stub: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/implicit/cpu/als.py(436): to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(68): load_artifacts

/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1/executor_model/1/model.py(71): initialize
E1028 15:34:10.075774 23778 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory
At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/implicit/cpu/als.py(436): to_gpu

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(68): load_artifacts

/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1/executor_model/1/model.py(71): initialize
I1028 15:34:10.075966 23778 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 15:34:10.076020 23778 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:34:10.076088 23778 server.cc:631]

+----------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model          | Version | Status                                                                                                                                                                                                                                    |

+----------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| executor_model | 1       | UNAVAILABLE: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory |

|                |         |                                                                                                                                                                                                                                           |

|                |         | At:                                                                                                                                                                                                                                       |

|                |         |   _cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit                                                                                                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/implicit/cpu/als.py(436): to_gpu                                                                                                           |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/implicit/gpu/matrix_factorization_base.py(212): load                                                                                       |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(68): load_artifacts                                                                                     |

|                |         |   /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1/executor_model/1/model.py(71): initialize                                                                                                                               |

+----------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:34:10.217717 23778 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 15:34:10.218609 23778 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:34:10.218647 23778 server.cc:262] Waiting for in-flight requests to complete.

I1028 15:34:10.218654 23778 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 15:34:10.218660 23778 server.cc:293] All models are stopped, unloading models

I1028 15:34:10.218666 23778 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1028 15:34:11.234596 23778 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

___________________ test_workflow_op_serving_python[parquet] ___________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7f76cc2a26a0>

engine = 'parquet'
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None
    with run_triton_server(tmpdir) as client:


      response = client.infer("0_transformworkflow", inputs, outputs=outputs)


tests/unit/systems/ops/nvtabular/test_ensemble.py:78:

.tox/test-cpu/lib/python3.8/site-packages/tritonclient/grpc/init.py:1431: in infer

raise_error_grpc(rpc_error)

rpc_error = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "Failed to process the reques...f-jenkins/pytest-5/test_workflow_op_serving_pytho0/0_transformworkflow/1/model.py(126): execute\n","grpc_status":13}"


def raise_error_grpc(rpc_error):


  raise get_error_grpc(rpc_error) from None


E       tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] Failed to process the request(s) for model instance '0_transformworkflow', message: AttributeError: 'NoneType' object has no attribute 'DataFrame'

E

E       At:

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(109): _array_to_cudf

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(74): convert_format

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(192): _transform_tensors

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(107): run_workflow

E         /tmp/pytest-of-jenkins/pytest-5/test_workflow_op_serving_pytho0/0_transformworkflow/1/model.py(126): execute
.tox/test-cpu/lib/python3.8/site-packages/tritonclient/grpc/init.py:62: InferenceServerException

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------

I1028 15:34:53.420954 24079 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f7dee000000' with size 268435456

I1028 15:34:53.421705 24079 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 15:34:53.426335 24079 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1028 15:34:56.489868 24079 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1028 15:34:58.331002 24079 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1028 15:34:58.331318 24079 model_lifecycle.cc:459] loading: ensemble_model:1

I1028 15:34:58.331664 24079 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1028 15:34:58.331773 24079 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 15:34:58.331861 24079 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:34:58.331928 24079 server.cc:631]

+---------------------+---------+--------+

| Model               | Version | Status |

+---------------------+---------+--------+

| 0_transformworkflow | 1       | READY  |

| ensemble_model      | 1       | READY  |

+---------------------+---------+--------+
I1028 15:34:58.447200 24079 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 15:34:58.448080 24079 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_workflow_op_serving_pytho0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:34:58.449620 24079 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1028 15:34:58.450137 24079 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1028 15:34:58.491318 24079 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1028 15:34:59.469588 24079 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:35:00.469776 24079 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:35:01.493368 24079 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

Failed to transform operator <nvtabular.ops.rename.Rename object at 0x7f2d4bff36d0>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py", line 183, in _transform_tensors

tensors, kind = convert_format(tensors, kind, workflow_node.inference_supports)

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py", line 74, in convert_format

return _array_to_cudf(tensors), Supports.GPU_DATAFRAME

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py", line 109, in _array_to_cudf

output = cudf.DataFrame()

AttributeError: 'NoneType' object has no attribute 'DataFrame'

1028 15:35:03.210926 24126 pb_stub.cc:777] Failed to process the request(s) for model '0_transformworkflow', message: AttributeError: 'NoneType' object has no attribute 'DataFrame'
At:

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(109): _array_to_cudf

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(74): convert_format

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(192): _transform_tensors

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(107): run_workflow

/tmp/pytest-of-jenkins/pytest-5/test_workflow_op_serving_pytho0/0_transformworkflow/1/model.py(126): execute
I1028 15:35:03.212084 24079 server.cc:262] Waiting for in-flight requests to complete.

I1028 15:35:03.212117 24079 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 15:35:03.212287 24079 server.cc:293] All models are stopped, unloading models

I1028 15:35:03.212309 24079 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1028 15:35:03.212319 24079 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1028 15:35:04.212411 24079 server.cc:300] Timeout 29: Found 1 live models and 0 in-flight non-inference requests

I1028 15:35:04.671917 24079 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1028 15:35:05.212540 24079 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

______________ test_workflow_tf_e2e_config_verification[parquet] _______________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0')

dataset = <merlin.io.dataset.Dataset object at 0x7f76cc1ddc40>

engine = 'parquet'
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_tf_e2e_config_verification(tmpdir, dataset, engine):
    # Create a Workflow
    schema = dataset.schema
    for name in ["x", "y", "id"]:
        dataset.schema.column_schemas[name] = dataset.schema.column_schemas[name].with_tags(
            [Tags.USER]
        )
    selector = ColumnSelector(["x", "y", "id"])

    workflow_ops = selector >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops["x_nvt"])
    workflow.fit(dataset)

    # Create Tensorflow Model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="x_nvt", dtype=tf.float64, shape=(1,)),
            tf.keras.layers.Dense(16, activation="relu"),
            tf.keras.layers.Dropout(0.2),
            tf.keras.layers.Dense(1, name="output"),
        ]
    )
    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    # Creating Triton Ensemble
    triton_chain = (
        selector >> TransformWorkflow(workflow, cats=["x_nvt"]) >> PredictTensorflow(model)
    )
    triton_ens = Ensemble(triton_chain, schema)

    # Creating Triton Ensemble Config
    ensemble_config, node_configs = triton_ens.export(str(tmpdir))

    config_path = tmpdir / ensemble_config.name / "config.pbtxt"

    # Checking Triton Ensemble Config
    with open(config_path, "rb") as f:
        config = model_config.ModelConfig()
        raw_config = f.read()
        parsed = text_format.Parse(raw_config, config)

        # The config file contents are correct
        assert parsed.name == "ensemble_model"
        assert parsed.platform == "ensemble"
        assert hasattr(parsed, "ensemble_scheduling")

    df = make_df({"x": [1.0, 2.0, 3.0], "y": [4.0, 5.0, 6.0], "id": [7, 8, 9]})

    request_schema = Schema([schema["x"], schema["y"], schema["id"]])

    output_columns = triton_ens.output_schema.column_names


  response = run_ensemble_on_tritonserver(


        str(tmpdir), request_schema, df, output_columns, ensemble_config.name
    )

tests/unit/systems/ops/tf/test_ensemble.py:108:

merlin/systems/triton/utils.py:139: in run_ensemble_on_tritonserver

response = send_triton_request(

merlin/systems/triton/utils.py:192: in send_triton_request

response = client.infer(triton_model, inputs, request_id=request_id, outputs=outputs)

.tox/test-cpu/lib/python3.8/site-packages/tritonclient/grpc/init.py:1431: in infer

raise_error_grpc(rpc_error)

rpc_error = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "in ensemble 'ensemble_model'...f-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/0_transformworkflow/1/model.py(126): execute\n","grpc_status":13}"


def raise_error_grpc(rpc_error):


  raise get_error_grpc(rpc_error) from None


E       tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '0_transformworkflow', message: AttributeError: 'NoneType' object has no attribute 'DataFrame'

E

E       At:

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(109): _array_to_cudf

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(74): convert_format

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(192): _transform_tensors

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(134): _transform_tensors

E         /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(107): run_workflow

E         /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/0_transformworkflow/1/model.py(126): execute
.tox/test-cpu/lib/python3.8/site-packages/tritonclient/grpc/init.py:62: InferenceServerException

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------

I1028 15:35:08.157103 24194 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fd954000000' with size 268435456

I1028 15:35:08.158090 24194 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1028 15:35:08.162868 24194 model_lifecycle.cc:459] loading: 1_predicttensorflowtriton:1

I1028 15:35:08.162911 24194 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1028 15:35:11.491602 24194 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1028 15:35:11.491649 24194 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1028 15:35:11.491656 24194 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1028 15:35:11.491662 24194 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1028 15:35:11.491703 24194 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1028 15:35:13.991624 24194 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 1_predicttensorflowtriton (version 1)

I1028 15:35:13.991895 24194 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

2022-10-28 15:35:13.992250: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/1_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:35:13.993537: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 15:35:13.993564: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/1_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:35:13.993671: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-28 15:35:14.046883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9001 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 15:35:14.077810: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-28 15:35:14.078574: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 15:35:14.153107: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/1_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:35:14.161709: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 169475 microseconds.

I1028 15:35:14.164450 24194 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 1_predicttensorflowtriton (GPU device 0)

2022-10-28 15:35:14.164813: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/1_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:35:14.165734: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-28 15:35:14.165754: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/1_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:35:14.167652: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9001 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-28 15:35:14.169776: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-28 15:35:14.193929: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/1_predicttensorflowtriton/1/model.savedmodel

2022-10-28 15:35:14.202630: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 37818 microseconds.

I1028 15:35:14.202839 24194 model_lifecycle.cc:693] successfully loaded '1_predicttensorflowtriton' version 1

I1028 15:35:14.203130 24194 model_lifecycle.cc:459] loading: ensemble_model:1

I1028 15:35:14.203338 24194 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1028 15:35:14.203436 24194 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+
I1028 15:35:14.203526 24194 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:35:14.203602 24194 server.cc:631]

+---------------------------+---------+--------+

| Model                     | Version | Status |

+---------------------------+---------+--------+

| 0_transformworkflow       | 1       | READY  |

| 1_predicttensorflowtriton | 1       | READY  |

| ensemble_model            | 1       | READY  |

+---------------------------+---------+--------+
I1028 15:35:14.276359 24194 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1028 15:35:14.277958 24194 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I1028 15:35:14.279797 24194 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1028 15:35:14.280092 24194 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1028 15:35:14.321267 24194 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1028 15:35:15.298930 24194 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:35:16.299141 24194 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1028 15:35:17.326288 24194 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

Failed to transform operator <nvtabular.ops.rename.Rename object at 0x7f83690c0730>

Traceback (most recent call last):

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py", line 183, in _transform_tensors

tensors, kind = convert_format(tensors, kind, workflow_node.inference_supports)

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py", line 74, in convert_format

return _array_to_cudf(tensors), Supports.GPU_DATAFRAME

File "/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py", line 109, in _array_to_cudf

output = cudf.DataFrame()

AttributeError: 'NoneType' object has no attribute 'DataFrame'

1028 15:35:17.936281 24241 pb_stub.cc:777] Failed to process the request(s) for model '0_transformworkflow', message: AttributeError: 'NoneType' object has no attribute 'DataFrame'
At:

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(109): _array_to_cudf

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/triton/conversions.py(74): convert_format

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(192): _transform_tensors

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(134): _transform_tensors

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/systems/workflow/base.py(107): run_workflow

/tmp/pytest-of-jenkins/pytest-5/test_workflow_tf_e2e_config_ve0/0_transformworkflow/1/model.py(126): execute
I1028 15:35:17.937572 24194 server.cc:262] Waiting for in-flight requests to complete.

I1028 15:35:17.937601 24194 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1028 15:35:17.937868 24194 server.cc:293] All models are stopped, unloading models

I1028 15:35:17.937874 24194 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1028 15:35:17.937899 24194 server.cc:300] Timeout 30: Found 3 live models and 0 in-flight non-inference requests

I1028 15:35:17.937984 24194 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1028 15:35:17.938080 24194 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1028 15:35:17.941672 24194 model_lifecycle.cc:578] successfully unloaded '1_predicttensorflowtriton' version 1

I1028 15:35:18.938024 24194 server.cc:300] Timeout 29: Found 1 live models and 0 in-flight non-inference requests

I1028 15:35:19.433997 24194 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1028 15:35:19.938152 24194 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

=============================== warnings summary ===============================

.tox/test-cpu/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
.tox/test-cpu/lib/python3.8/site-packages/faiss/loader.py:28

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
.tox/test-cpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/pandas/core/indexing.py:1732: SettingWithCopyWarning:

A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

self._setitem_single_block(indexer, value, name)
tests/unit/test_export.py: 2 warnings

tests/unit/systems/dag/test_op_runner.py: 4 warnings

tests/unit/systems/ops/fil/test_ensemble.py: 1 warning

tests/unit/systems/ops/fil/test_forest.py: 2 warnings

tests/unit/systems/ops/nvtabular/test_ensemble.py: 1 warning

tests/unit/systems/ops/nvtabular/test_op.py: 2 warnings

tests/unit/systems/ops/tf/test_ensemble.py: 10 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/merlin/io/dataset.py:251: UserWarning: Initializing an NVTabular Dataset in CPU mode.This is an experimental feature with extremely limited support!

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_ensemble.py::test_workflow_with_forest_inference

tests/unit/systems/ops/fil/test_forest.py::test_ensemble

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/lib/python3.8/site-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     71    18%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     11    85%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1134    57%
=========================== short test summary info ============================

SKIPPED [1] tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py:8: could not import 'cudf': No module named 'cudf'

SKIPPED [1] tests/unit/systems/dag/test_executors.py:64: unable to find GPU

SKIPPED [1] tests/unit/systems/dag/test_executors.py:97: unable to find GPU

======= 6 failed, 82 passed, 3 skipped, 40 warnings in 599.51s (0:09:59) =======

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-cpu/bin/python -m pytest --cov-report term --cov=merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

python3.8: commands succeeded

ERROR:   test-gpu: commands failed

ERROR:   test-cpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins12833372921357183686.sh

jperez999 · 2022-10-31T14:43:05Z

rerun tests

nvidia-merlin-bot · 2022-10-31T14:43:22Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/471/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins12421894130188347858.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
python3.8 recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/python3.8
ERROR: invocation failed (exit code -15), logfile: /var/jenkins_home/workspace/merlin_systems/systems/.tox/python3.8/log/python3.8-0.log
================================== log start ===================================
=================================== log end ====================================

ERROR: InvocationError for command /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 python3.8 (exited with code -15 (SIGTERM)) (exited with code -15)

test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu

ERROR: Got SIGTERM, handling it as a KeyboardInterrupt

___________________________________ summary ____________________________________

ERROR:   python3.8: InvocationError for command /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 python3.8 (exited with code -15 (SIGTERM)) (exited with code -15)

ERROR:   test-gpu: keyboardinterrupt

ERROR:   test-cpu: undefined

Build was aborted

Aborted by �[8mha:////4I6AZwo/1Z8Fal8AhZTEatjIwqNwCcqT21311HdysuK+AAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzWEgZu/dLi1CL9xJTczDwAj6GcLcAAAAA=�[0madmin

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins4016619678607012208.sh

jperez999 · 2022-10-31T14:43:31Z

rerun tests

nvidia-merlin-bot · 2022-10-31T14:45:16Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/472/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins16883494842724491473.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
python3.8 create: /var/jenkins_home/workspace/merlin_systems/systems/.tox/python3.8
python3.8 inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
python3.8 installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.4,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,bleach==5.0.1,boto3==1.24.75,botocore==1.28.4,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular==1.4.0+8.g95e12d347,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
python3.8 run-test-pre: PYTHONHASHSEED='1532553771'
python3.8 run-test: commands[0] | pip install --upgrade pip
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./.tox/python3.8/lib/python3.8/site-packages (22.2.2)
Collecting pip
  Downloading pip-22.3-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 14.9 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.2
    Uninstalling pip-22.2.2:
      Successfully uninstalled pip-22.2.2
Successfully installed pip-22.3
python3.8 run-test: commands[1] | pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (2.28.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0)
Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.20.3)
Requirement already satisfied: tqdm>=4.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)
Requirement already satisfied: numba>=0.54 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.56.3)
Requirement already satisfied: fsspec==2022.5.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)
Requirement already satisfied: packaging in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (10.0.0)
Requirement already satisfied: dask>=2022.3.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.1)
Requirement already satisfied: betterproto<2.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)
Requirement already satisfied: scipy in ./.tox/python3.8/lib/python3.8/site-packages (from nvtabular>=1.0.0->merlin-systems==0.7.0+19.g736699e) (1.9.3)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2022.9.24)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2.1.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (1.26.12)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (3.4)
Requirement already satisfied: stringcase in ./.tox/python3.8/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)
Requirement already satisfied: grpclib in ./.tox/python3.8/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)
Requirement already satisfied: click>=7.0 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)
Requirement already satisfied: toolz>=0.8.2 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0)
Requirement already satisfied: partd>=0.3.10 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)
Requirement already satisfied: msgpack>=0.6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)
Requirement already satisfied: jinja2 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)
Requirement already satisfied: locket>=1.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.0)
Requirement already satisfied: psutil>=5.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.9.3)
Requirement already satisfied: zict>=0.1.3 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)
Requirement already satisfied: tornado<6.2,>=6.0.3 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)
Requirement already satisfied: importlib-metadata in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.0.0)
Requirement already satisfied: setuptools in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)
Requirement already satisfied: numpy<1.24,>=1.18 in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.23.4)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.39.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/python3.8/lib/python3.8/site-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in ./.tox/python3.8/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5)
Requirement already satisfied: python-dateutil>=2.7.3 in ./.tox/python3.8/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/python3.8/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/python3.8/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.56.4)
Requirement already satisfied: six>=1.5 in ./.tox/python3.8/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.16.0)
Requirement already satisfied: heapdict in ./.tox/python3.8/lib/python3.8/site-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/python3.8/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)
Requirement already satisfied: multidict in ./.tox/python3.8/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)
Requirement already satisfied: zipp>=0.5 in ./.tox/python3.8/lib/python3.8/site-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.10.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.tox/python3.8/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=97669 sha256=9a89dc2b04fe9b43d0da36d4755ddafc06479ab037067ab92ffb9bbfa111af8a
  Stored in directory: /tmp/pip-ephem-wheel-cache-f6r_20ae/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+19.g736699e
    Uninstalling merlin-systems-0.7.0+19.g736699e:
      Successfully uninstalled merlin-systems-0.7.0+19.g736699e
Successfully installed merlin-systems-0.7.0+19.g736699e
test-gpu create: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
ERROR: cowardly refusing to delete `envdir` (it does not look like a virtualenv): /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
___________________________________ summary ____________________________________
  python3.8: commands succeeded
ERROR:   test-gpu: error
ERROR:   test-cpu: undefined
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_systems] $ /bin/bash /tmp/jenkins2930827741072359916.sh

jperez999 · 2022-10-31T14:54:27Z

rerun tests

nvidia-merlin-bot · 2022-10-31T14:56:14Z

Click to view CI Results

GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
GitHub pull request #219 of commit 736699efa81531b4fe93b81d835c2bc6c6505f82, no merge conflicts.
Running as SYSTEM
Setting status of 736699efa81531b4fe93b81d835c2bc6c6505f82 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/473/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 736699efa81531b4fe93b81d835c2bc6c6505f82^{commit} # timeout=10
Checking out Revision 736699efa81531b4fe93b81d835c2bc6c6505f82 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
Commit message: "Remove `exportable_backends` from `PredictTensorflow`, add docstrings"
 > git rev-list --no-walk 736699efa81531b4fe93b81d835c2bc6c6505f82 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins10603366585879653653.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
python3.8 recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/python3.8
python3.8 inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+19.g736699e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
python3.8 installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.4,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,bleach==5.0.1,boto3==1.24.75,botocore==1.28.4,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+19.g736699e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular==1.4.0+8.g95e12d347,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
python3.8 run-test-pre: PYTHONHASHSEED='2057134052'
python3.8 run-test: commands[0] | pip install --upgrade pip
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./.tox/python3.8/lib/python3.8/site-packages (22.2.2)
Collecting pip
  Downloading pip-22.3-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 12.9 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.2
    Uninstalling pip-22.2.2:
      Successfully uninstalled pip-22.2.2
Successfully installed pip-22.3
python3.8 run-test: commands[1] | pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (2.28.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (0.8.0)
Requirement already satisfied: nvtabular>=1.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-systems==0.7.0+19.g736699e) (1.5.0)
Requirement already satisfied: dask>=2022.3.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.1)
Requirement already satisfied: pyarrow>=5.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (10.0.0)
Requirement already satisfied: distributed>=2022.3.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.10.1)
Requirement already satisfied: protobuf>=3.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.20.3)
Requirement already satisfied: betterproto<2.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5.0)
Requirement already satisfied: numba>=0.54 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.56.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.10.0)
Requirement already satisfied: packaging in ./.tox/python3.8/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (21.3)
Requirement already satisfied: scipy in ./.tox/python3.8/lib/python3.8/site-packages (from nvtabular>=1.0.0->merlin-systems==0.7.0+19.g736699e) (1.9.3)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (1.26.12)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2022.9.24)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/python3.8/lib/python3.8/site-packages (from requests<3,>=2.10->merlin-systems==0.7.0+19.g736699e) (2.1.1)
Requirement already satisfied: grpclib in ./.tox/python3.8/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.4.3)
Requirement already satisfied: stringcase in ./.tox/python3.8/lib/python3.8/site-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)
Requirement already satisfied: toolz>=0.8.2 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.12.0)
Requirement already satisfied: click>=7.0 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (8.1.3)
Requirement already satisfied: pyyaml>=5.3.1 in ./.tox/python3.8/lib/python3.8/site-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0)
Requirement already satisfied: tblib>=1.6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.7.0)
Requirement already satisfied: jinja2 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.1.2)
Requirement already satisfied: tornado<6.2,>=6.0.3 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.1)
Requirement already satisfied: psutil>=5.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.9.3)
Requirement already satisfied: msgpack>=0.6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.4.0)
Requirement already satisfied: locket>=1.0.0 in ./.tox/python3.8/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (0.39.1)
Requirement already satisfied: setuptools in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (65.4.1)
Requirement already satisfied: numpy<1.24,>=1.18 in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.23.4)
Requirement already satisfied: importlib-metadata in ./.tox/python3.8/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (5.0.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/python3.8/lib/python3.8/site-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in ./.tox/python3.8/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in ./.tox/python3.8/lib/python3.8/site-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2022.5)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/python3.8/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.56.4)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/python3.8/lib/python3.8/site-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.3.0)
Requirement already satisfied: six>=1.5 in ./.tox/python3.8/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.16.0)
Requirement already satisfied: heapdict in ./.tox/python3.8/lib/python3.8/site-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/python3.8/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.1.0)
Requirement already satisfied: multidict in ./.tox/python3.8/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.2)
Requirement already satisfied: zipp>=0.5 in ./.tox/python3.8/lib/python3.8/site-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (3.10.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/python3.8/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.tox/python3.8/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/python3.8/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+19.g736699e) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+19.g736699e-py3-none-any.whl size=91040 sha256=1e448067edf228ab341dbc065750c604d6313dfd36a583376ee44f45aa9b9c93
  Stored in directory: /tmp/pip-ephem-wheel-cache-3oxhho45/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+19.g736699e
    Uninstalling merlin-systems-0.7.0+19.g736699e:
      Successfully uninstalled merlin-systems-0.7.0+19.g736699e
Successfully installed merlin-systems-0.7.0+19.g736699e
test-gpu create: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
ERROR: cowardly refusing to delete `envdir` (it does not look like a virtualenv): /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
___________________________________ summary ____________________________________
  python3.8: commands succeeded
ERROR:   test-gpu: error
ERROR:   test-cpu: undefined
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[merlin_systems] $ /bin/bash /tmp/jenkins498203670100814177.sh

jperez999 · 2022-10-31T15:27:10Z

rerun tests

jperez999 · 2022-10-31T16:23:02Z

rerun tests

karlhigley · 2022-10-31T21:02:14Z

@jperez999 tells me that the main thread of PyTest grabs half the GPU memory via configure_tensorflow when running the examples, and then grabs another half when running the normal tests.

jperez999 · 2022-11-01T15:58:59Z

rerun tests

jperez999 · 2022-11-01T16:00:32Z

rerun tests

nvidia-merlin-bot · 2022-11-01T16:32:27Z

Click to view CI Results

GitHub pull request #219 of commit 662a1ea64c43262803d3aaebad6206b8e110458f, no merge conflicts.
GitHub pull request #219 of commit 662a1ea64c43262803d3aaebad6206b8e110458f, no merge conflicts.
Running as SYSTEM
Setting status of 662a1ea64c43262803d3aaebad6206b8e110458f to PENDING with url https://10.20.13.93:8080/job/merlin_systems/480/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/219/*:refs/remotes/origin/pr/219/* # timeout=10
 > git rev-parse 662a1ea64c43262803d3aaebad6206b8e110458f^{commit} # timeout=10
Checking out Revision 662a1ea64c43262803d3aaebad6206b8e110458f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 662a1ea64c43262803d3aaebad6206b8e110458f # timeout=10
Commit message: "Merge branch 'main' into triton-op-tensorflow-runtime"
 > git rev-list --no-walk 1bd200d5705793fd0281a19c05ba4261eb671d1d # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins648291388071844081.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.7.0+21.g662a1ea.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.26.5,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.28.5,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.0.0,cloudpickle==2.2.0,cmaes==0.8.2,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.3,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+21.g662a1ea,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.42,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.0,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1216772782'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-buwqcrji
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-buwqcrji
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit eda153c663aa864da66927c7a0a9d4e64c073120
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (3.19.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (0.55.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (1.2.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.8.0+4.geda153c) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.8.0+4.geda153c) (2022.5.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (0.4.3)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (65.4.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.8.0+4.geda153c) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.8.0+4.geda153c) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.8.0+4.geda153c) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.8.0+4.geda153c) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.8.0+4.geda153c) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.8.0+4.geda153c) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.8.0+4.geda153c-py3-none-any.whl size=118258 sha256=09eadfa3861cc9b625969b5b01aedf7f67f294f65112892f6ac7ff9b1f739eed
  Stored in directory: /tmp/pip-ephem-wheel-cache-2r7dep3c/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.8.0+4.geda153c
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.g662a1ea) (1.1.1)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+21.g662a1ea) (2.22.0)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+21.g662a1ea) (0.8.0+4.geda153c)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.10.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (3.19.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (0.55.1)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (7.0.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (21.3)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.2.5)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (4.64.1)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.3.5)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2022.5.0)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (0.4.3)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.2.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (8.1.3)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.7.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (6.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (5.8.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2.0.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2.4.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (3.1.2)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (65.4.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+21.g662a1ea) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.7.0+21.g662a1ea-py3-none-any.whl size=94186 sha256=daf70613d7084c5cb8698a4fc640f3ccc0544ed429e405cde2d3562b9cce070c

Stored in directory: /tmp/pip-ephem-wheel-cache-l9khpbr4/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.7.0+21.g662a1ea

Uninstalling merlin-systems-0.7.0+21.g662a1ea:

Successfully uninstalled merlin-systems-0.7.0+21.g662a1ea

Successfully installed merlin-systems-0.7.0+21.g662a1ea
[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0

collected 92 items
tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ...                             [  9%]

tests/unit/systems/dag/test_graph.py ..                                  [ 11%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 14%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 18%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 20%]

tests/unit/systems/dag/runtimes/test_triton.py .....                     [ 26%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 27%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 35%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 71%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 81%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 82%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 84%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 89%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]
=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)
.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)
../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(
../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(
tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)
tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(
tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover
merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             3      0   100%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                         9      0   100%

merlin/systems/dag/ops/compat.py                          24      8    67%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          126      7    94%

merlin/systems/dag/ops/fil.py                            223     34    85%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         89     12    87%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        72     26    64%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      61     12    80%

merlin/systems/dag/runtimes/triton/runtime.py            140      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         47     17    64%

merlin/systems/triton/conversions.py                      87     73    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            41     41     0%

merlin/systems/triton/models/oprunner_model.py            41     41     0%

merlin/systems/triton/models/pytorch_model.py            104    104     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     14    81%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%
TOTAL                                                   2610   1139    56%
================= 92 passed, 56 warnings in 846.61s (0:14:06) ==================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins15890624016107827253.sh

oliverholworthy and others added 11 commits October 27, 2022 19:16

Split out PredictTensorflow into two Operators

27a6e42

Co-authored-by: Karl Higley <[email protected]> Co-authored-by: Julio Perez <[email protected]>

raise NotImplementedError in InferenceOperator export abstractmethod

4f832a3

Co-authored-by: Julio Perez <[email protected]> Co-authored-by: Karl Higley <[email protected]>

Move Tensorflow Triton op to Triton Runtime module

27e6459

Co-authored-by: Julio Perez <[email protected]> Co-authored-by: Karl Higley <[email protected]>

Move triton runtime code to subdirectory

56e0d8f

Co-authored-by: Julio Perez <[email protected]> Co-authored-by: Karl Higley <[email protected]>

Use importlib.resources to find oprunner model

4e57c44

Use importlib.resources to find workflow_model.py

29e8e1c

Move TritonOperator to operator module

35473e4

Move _tf_model_name from base tensorflow op to triton op

75f37f9

Construct InferenceResponse correctly when an error occurs

fe923c5

Add docstrings to TritonOperator

0dd6c5d

Remove exportable_backends from PredictTensorflow, add docstrings

736699e

oliverholworthy added the chore Maintenance for the repository label Oct 27, 2022

oliverholworthy added this to the Merlin 22.11 milestone Oct 27, 2022

oliverholworthy assigned karlhigley, oliverholworthy and jperez999 Oct 27, 2022

karlhigley approved these changes Oct 27, 2022

View reviewed changes

oliverholworthy commented Oct 27, 2022

View reviewed changes

karlhigley removed their assignment Oct 28, 2022

jperez999 mentioned this pull request Nov 1, 2022

add env variable to handle gpu memory allocation #221

Merged

Merge branch 'main' into triton-op-tensorflow-runtime

662a1ea

jperez999 merged commit d2737da into NVIDIA-Merlin:main Nov 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor TensorFlow operator into base and Triton runtime operator #219

Refactor TensorFlow operator into base and Triton runtime operator #219

oliverholworthy commented Oct 27, 2022

github-actions bot commented Oct 27, 2022

karlhigley Oct 27, 2022

oliverholworthy Oct 27, 2022

oliverholworthy commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

oliverholworthy commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

karlhigley commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

jperez999 commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

jperez999 commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

jperez999 commented Oct 31, 2022

nvidia-merlin-bot commented Oct 31, 2022

jperez999 commented Oct 31, 2022

nvidia-merlin-bot commented Oct 31, 2022

jperez999 commented Oct 31, 2022

nvidia-merlin-bot commented Oct 31, 2022

jperez999 commented Oct 31, 2022

jperez999 commented Oct 31, 2022

karlhigley commented Oct 31, 2022

jperez999 commented Nov 1, 2022

jperez999 commented Nov 1, 2022

nvidia-merlin-bot commented Nov 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

Refactor TensorFlow operator into base and Triton runtime operator #219

Refactor TensorFlow operator into base and Triton runtime operator #219

Conversation

oliverholworthy commented Oct 27, 2022

github-actions bot commented Oct 27, 2022

Documentation preview

karlhigley Oct 27, 2022

Choose a reason for hiding this comment

oliverholworthy Oct 27, 2022

Choose a reason for hiding this comment

Example

oliverholworthy commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

oliverholworthy commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

karlhigley commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

jperez999 commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

jperez999 commented Oct 28, 2022

nvidia-merlin-bot commented Oct 28, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

jperez999 commented Oct 31, 2022

nvidia-merlin-bot commented Oct 31, 2022

jperez999 commented Oct 31, 2022

nvidia-merlin-bot commented Oct 31, 2022

jperez999 commented Oct 31, 2022

nvidia-merlin-bot commented Oct 31, 2022

jperez999 commented Oct 31, 2022

jperez999 commented Oct 31, 2022

karlhigley commented Oct 31, 2022

jperez999 commented Nov 1, 2022

jperez999 commented Nov 1, 2022

nvidia-merlin-bot commented Nov 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover