Skip to content

Rework Systems operators to use DictArray and LocalExecutor from Merlin Core #204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jperez999 merged 15 commits into from
Oct 21, 2022
Merged

Rework Systems operators to use DictArray and LocalExecutor from Merlin Core #204

jperez999 merged 15 commits into from
Oct 21, 2022

Conversation

@karlhigley
Copy link
Contributor

@karlhigley karlhigley commented Sep 15, 2022 •
edited
Loading

Depends on NVIDIA-Merlin/core#139 and NVIDIA-Merlin/core#146

We've had the concept of an InferenceDataFrame for a while, but it's really just a wrapper around a dictionary of arrays. That data structure is useful in a bunch of places, so this swaps out InferenceDataFrame for a similar class from Merlin Core called DictArray.

@karlhigley karlhigley added clean up chore Maintenance for the repository labels Sep 15, 2022
@karlhigley karlhigley added this to the Merlin 22.10 milestone Sep 15, 2022
@karlhigley karlhigley self-assigned this Sep 15, 2022
@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Sep 15, 2022

Click to view CI Results
GitHub pull request #204 of commit e6eb867028fd042afe77a8b580261b078b1eae2e, no merge conflicts.
Running as SYSTEM
Setting status of e6eb867028fd042afe77a8b580261b078b1eae2e to PENDING with url https://10.20.13.93:8080/job/merlin_systems/396/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse e6eb867028fd042afe77a8b580261b078b1eae2e^{commit} # timeout=10
Checking out Revision e6eb867028fd042afe77a8b580261b078b1eae2e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e6eb867028fd042afe77a8b580261b078b1eae2e # timeout=10
Commit message: "Fix Triton model param handling in `PipelineableInferenceOperator`"
 > git rev-list --no-walk 310597f4ec4e423326ef8b6f235353c3612f3545 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins17533510124154292747.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+6.ge6eb867.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.7,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,attrs==22.1.0,awscli==1.25.73,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.51,botocore==1.27.72,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,clang==5.0,click==8.1.3,cloudpickle==2.1.0,colorama==0.4.4,coverage==6.4.4,cuda-python==11.7.1,cudf==22.4.0,cupy-cuda116==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dask-cuda==22.4.0,dask-cudf==22.4.0,dbus-python==1.2.16,debugpy==1.6.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.3.0,distro==1.7.0,dm-tree==0.1.7,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==0.10.0,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.82.0,fastavro==1.6.0,fastcore==1.5.24,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.0,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.0,google-auth==2.11.0,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.2,grpcio==1.41.0,grpcio-channelz==1.47.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.4.0,hugectr2onnx==0.0.0,huggingface-hub==0.8.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.0,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.1,ipython==8.4.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.1.0,json5==0.9.9,jsonschema==4.9.1,jupyter-cache==0.4.3,jupyter-client==7.3.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyterlab==3.4.5,jupyterlab-pygments==0.2.2,jupyterlab-server==2.15.0,jupyterlab-widgets==1.1.0,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.0,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.5.3,matplotlib-inline==0.1.3,mdit-py-plugins==0.2.8,merlin-core==0+untagged.112.gfbfcaf3,merlin-models==0.6.0+45.g5a345d9c1,merlin-systems==0.5.0+6.ge6eb867,mistune==0.8.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.6,nbconvert==6.5.3,nbdime==3.1.1,nbformat==5.4.0,nest-asyncio==1.5.5,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.0,numpy==1.21.5,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.3.3+15.g16e4e34e9),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.0,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.30,proto-plus==1.19.6,protobuf==3.19.4,psutil==5.9.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==6.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.12.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.7,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==23.2.1,regex==2022.7.25,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rmm==21.12.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.0,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.4,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.2.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.4.0,starlette==0.19.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.26.0,tensorflow-metadata==1.9.0,termcolor==1.1.0,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.25.1,tqdm==4.64.0,traitlets==5.3.0,transformers==4.12.0,transformers4rec==0.1.11+10.g21a2a836a,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.22.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.11,uvicorn==0.18.3,uvloop==0.16.0,versioneer==0.20,virtualenv==20.16.4,wandb==0.13.1,watchfiles==0.16.1,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.3.3,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='943676117'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 6 items / 9 errors / 4 skipped

==================================== ERRORS ====================================

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:28: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.dag import ColumnSelector, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:28: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

import merlin.systems.dag.ops.fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.dag import ColumnSelector, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:28: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:28: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                  78     70    10%

merlin/systems/dag/node.py                      20     20     0%

merlin/systems/dag/op_runner.py                 21     21     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 60     60     0%

merlin/systems/dag/ops/feast.py                121    121     0%

merlin/systems/dag/ops/fil.py                  212    205     3%

merlin/systems/dag/ops/implicit.py              52     52     0%

merlin/systems/dag/ops/operator.py              68     58    15%

merlin/systems/dag/ops/pytorch.py               57     57     0%

merlin/systems/dag/ops/session_filter.py        43     43     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py            81     69    15%

merlin/systems/dag/ops/unroll_features.py       49     49     0%

merlin/systems/dag/ops/workflow.py              25     25     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2036   1759    14%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

!!!!!!!!!!!!!!!!!!! Interrupted: 9 errors during collection !!!!!!!!!!!!!!!!!!!!

================== 4 skipped, 36 warnings, 9 errors in 5.10s ===================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins17869355340019852631.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Sep 19, 2022

Click to view CI Results
GitHub pull request #204 of commit 6c75fdbeb7bf1c684558b0d087a29adcd3b64921, no merge conflicts.
Running as SYSTEM
Setting status of 6c75fdbeb7bf1c684558b0d087a29adcd3b64921 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/397/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 6c75fdbeb7bf1c684558b0d087a29adcd3b64921^{commit} # timeout=10
Checking out Revision 6c75fdbeb7bf1c684558b0d087a29adcd3b64921 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6c75fdbeb7bf1c684558b0d087a29adcd3b64921 # timeout=10
Commit message: "Run the filter op in `LocalExecutor` in a `TritonPythonModel`"
 > git rev-list --no-walk e6eb867028fd042afe77a8b580261b078b1eae2e # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins5842763780289934621.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+8.g6c75fdb.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.7,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,attrs==22.1.0,awscli==1.25.73,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.51,botocore==1.27.72,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,clang==5.0,click==8.1.3,cloudpickle==2.1.0,colorama==0.4.4,coverage==6.4.4,cuda-python==11.7.1,cudf==22.4.0,cupy-cuda116==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dask-cuda==22.4.0,dask-cudf==22.4.0,dbus-python==1.2.16,debugpy==1.6.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.3.0,distro==1.7.0,dm-tree==0.1.7,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==0.10.0,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.82.0,fastavro==1.6.0,fastcore==1.5.24,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.0,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.0,google-auth==2.11.0,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.2,grpcio==1.41.0,grpcio-channelz==1.47.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.4.0,hugectr2onnx==0.0.0,huggingface-hub==0.8.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.0,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.1,ipython==8.4.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.1.0,json5==0.9.9,jsonschema==4.9.1,jupyter-cache==0.4.3,jupyter-client==7.3.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyterlab==3.4.5,jupyterlab-pygments==0.2.2,jupyterlab-server==2.15.0,jupyterlab-widgets==1.1.0,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.0,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.5.3,matplotlib-inline==0.1.3,mdit-py-plugins==0.2.8,merlin-core==0+untagged.112.gfbfcaf3,merlin-models==0.6.0+45.g5a345d9c1,merlin-systems==0.5.0+8.g6c75fdb,mistune==0.8.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.6,nbconvert==6.5.3,nbdime==3.1.1,nbformat==5.4.0,nest-asyncio==1.5.5,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.0,numpy==1.21.5,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.3.3+15.g16e4e34e9),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.0,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.30,proto-plus==1.19.6,protobuf==3.19.4,psutil==5.9.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==6.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.12.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.7,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==23.2.1,regex==2022.7.25,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rmm==21.12.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.0,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.4,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.2.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.4.0,starlette==0.19.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.26.0,tensorflow-metadata==1.9.0,termcolor==1.1.0,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.25.1,tqdm==4.64.0,traitlets==5.3.0,transformers==4.12.0,transformers4rec==0.1.11+10.g21a2a836a,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.22.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.11,uvicorn==0.18.3,uvloop==0.16.0,versioneer==0.20,virtualenv==20.16.4,wandb==0.13.1,watchfiles==0.16.1,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.3.3,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3672008054'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 6 items / 11 errors / 4 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:26: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.dag import ColumnSelector, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

import merlin.systems.dag.ops.fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.dag import ColumnSelector, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.dag import BaseOperator, DictArray  # noqa

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 104     90    13%

merlin/systems/dag/node.py                      20     20     0%

merlin/systems/dag/op_runner.py                 24     24     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 60     60     0%

merlin/systems/dag/ops/feast.py                121    121     0%

merlin/systems/dag/ops/fil.py                  212    205     3%

merlin/systems/dag/ops/implicit.py              52     52     0%

merlin/systems/dag/ops/operator.py              68     58    15%

merlin/systems/dag/ops/pytorch.py               57     57     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py            81     69    15%

merlin/systems/dag/ops/unroll_features.py       49     49     0%

merlin/systems/dag/ops/workflow.py              25     25     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2066   1783    14%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!

================== 4 skipped, 36 warnings, 11 errors in 5.25s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins3198920127711845033.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Sep 19, 2022

Click to view CI Results
GitHub pull request #204 of commit 17f55cc343ac4130f087db192d6fdfb544831249, no merge conflicts.
Running as SYSTEM
Setting status of 17f55cc343ac4130f087db192d6fdfb544831249 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/398/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 17f55cc343ac4130f087db192d6fdfb544831249^{commit} # timeout=10
Checking out Revision 17f55cc343ac4130f087db192d6fdfb544831249 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 17f55cc343ac4130f087db192d6fdfb544831249 # timeout=10
Commit message: "(WIP) Update transform methods"
 > git rev-list --no-walk 6c75fdbeb7bf1c684558b0d087a29adcd3b64921 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins16760520706394828778.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+9.g17f55cc.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.7,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,attrs==22.1.0,awscli==1.25.73,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.51,botocore==1.27.72,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,clang==5.0,click==8.1.3,cloudpickle==2.1.0,colorama==0.4.4,coverage==6.4.4,cuda-python==11.7.1,cudf==22.4.0,cupy-cuda116==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dask-cuda==22.4.0,dask-cudf==22.4.0,dbus-python==1.2.16,debugpy==1.6.2,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.3.0,distro==1.7.0,dm-tree==0.1.7,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==0.10.0,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.82.0,fastavro==1.6.0,fastcore==1.5.24,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.0,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.0,google-auth==2.11.0,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.2,grpcio==1.41.0,grpcio-channelz==1.47.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.4.0,hugectr2onnx==0.0.0,huggingface-hub==0.8.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.0,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.1,ipython==8.4.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.1.0,json5==0.9.9,jsonschema==4.9.1,jupyter-cache==0.4.3,jupyter-client==7.3.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyterlab==3.4.5,jupyterlab-pygments==0.2.2,jupyterlab-server==2.15.0,jupyterlab-widgets==1.1.0,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.0,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.5.3,matplotlib-inline==0.1.3,mdit-py-plugins==0.2.8,merlin-core==0+untagged.112.gfbfcaf3,merlin-models==0.6.0+45.g5a345d9c1,merlin-systems==0.5.0+9.g17f55cc,mistune==0.8.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.6,nbconvert==6.5.3,nbdime==3.1.1,nbformat==5.4.0,nest-asyncio==1.5.5,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.0,numpy==1.21.5,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.3.3+15.g16e4e34e9),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.0,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.30,proto-plus==1.19.6,protobuf==3.19.4,psutil==5.9.1,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==6.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.12.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.7,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==23.2.1,regex==2022.7.25,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rmm==21.12.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.0,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.4,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.2.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.4.0,starlette==0.19.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.26.0,tensorflow-metadata==1.9.0,termcolor==1.1.0,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.25.1,tqdm==4.64.0,traitlets==5.3.0,transformers==4.12.0,transformers4rec==0.1.11+10.g21a2a836a,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.22.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.11,uvicorn==0.18.3,uvloop==0.16.0,versioneer==0.20,virtualenv==20.16.4,wandb==0.13.1,watchfiles==0.16.1,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.3.3,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='639805855'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 6 items / 11 errors / 4 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:26: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 104     90    13%

merlin/systems/dag/node.py                      20     20     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 61     61     0%

merlin/systems/dag/ops/feast.py                122    122     0%

merlin/systems/dag/ops/fil.py                  216    209     3%

merlin/systems/dag/ops/implicit.py              53     53     0%

merlin/systems/dag/ops/operator.py              70     60    14%

merlin/systems/dag/ops/pytorch.py               57     57     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      52     52     0%

merlin/systems/dag/ops/tensorflow.py            86     74    14%

merlin/systems/dag/ops/workflow.py              25     25     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2034   1751    14%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!

================== 4 skipped, 36 warnings, 11 errors in 4.81s ==================

/usr/local/lib/python3.8/dist-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/dag/ops/unroll_features.py' (couldnt-parse)

coverage._warn(msg, slug="couldnt-parse")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins293271482406581028.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Sep 20, 2022

Click to view CI Results
GitHub pull request #204 of commit 06fc234c98d982066cbae55e851eb84a18ab99b1, no merge conflicts.
Running as SYSTEM
Setting status of 06fc234c98d982066cbae55e851eb84a18ab99b1 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/399/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 06fc234c98d982066cbae55e851eb84a18ab99b1^{commit} # timeout=10
Checking out Revision 06fc234c98d982066cbae55e851eb84a18ab99b1 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 06fc234c98d982066cbae55e851eb84a18ab99b1 # timeout=10
Commit message: "Update TODO list"
 > git rev-list --no-walk 17f55cc343ac4130f087db192d6fdfb544831249 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins10660284836720595816.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+11.g06fc234.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.77,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.76,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.4.4,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.2,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+6.ga00b40ba3,merlin-systems==0.5.0+11.g06fc234,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.1.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2080873586'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 6 items / 11 errors / 4 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:26: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:35: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 104     90    13%

merlin/systems/dag/node.py                      20     20     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 61     61     0%

merlin/systems/dag/ops/feast.py                122    122     0%

merlin/systems/dag/ops/fil.py                  216    209     3%

merlin/systems/dag/ops/implicit.py              53     53     0%

merlin/systems/dag/ops/operator.py              70     60    14%

merlin/systems/dag/ops/pytorch.py               57     57     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      52     52     0%

merlin/systems/dag/ops/tensorflow.py           127    117     8%

merlin/systems/dag/ops/workflow.py              25     25     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2075   1794    14%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!

================== 4 skipped, 36 warnings, 11 errors in 5.16s ==================

/usr/local/lib/python3.8/dist-packages/coverage/report.py:81: CoverageWarning: Couldn't parse Python file '/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/dag/ops/unroll_features.py' (couldnt-parse)

coverage._warn(msg, slug="couldnt-parse")

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins10508599641261452100.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Sep 26, 2022

Click to view CI Results
GitHub pull request #204 of commit 6e39c19747a40910cf4c3e2b4bfc8ba8ce824f93, no merge conflicts.
Running as SYSTEM
Setting status of 6e39c19747a40910cf4c3e2b4bfc8ba8ce824f93 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/401/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 6e39c19747a40910cf4c3e2b4bfc8ba8ce824f93^{commit} # timeout=10
Checking out Revision 6e39c19747a40910cf4c3e2b4bfc8ba8ce824f93 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6e39c19747a40910cf4c3e2b4bfc8ba8ce824f93 # timeout=10
Commit message: "Linter fixes"
 > git rev-list --no-walk 935c088b088768943733a58566c0d492ee49ce19 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins712096128434959780.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+22.g6e39c19.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.81,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.80,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.4.4,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+22.g6e39c19,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.1,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='886102185'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:26: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_ensemble.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_ensemble.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 128    112    12%

merlin/systems/dag/node.py                      20     20     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 66     66     0%

merlin/systems/dag/ops/feast.py                124    124     0%

merlin/systems/dag/ops/fil.py                  218    211     3%

merlin/systems/dag/ops/implicit.py              55     55     0%

merlin/systems/dag/ops/operator.py              70     60    14%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      52     52     0%

merlin/systems/dag/ops/tensorflow.py           131    121     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2258   1975    13%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:35: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 5.20s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins948432215355933803.sh

@karlhigley karlhigley changed the title Rework Systems operators to use DictArray from Merlin Core Rework Systems operators to use DictArray and LocalExecutor from Merlin Core Sep 27, 2022
@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Sep 28, 2022

Click to view CI Results
GitHub pull request #204 of commit d03b49f46de81ed81a11a47fb381b88192c7f300, no merge conflicts.
Running as SYSTEM
Setting status of d03b49f46de81ed81a11a47fb381b88192c7f300 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/402/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse d03b49f46de81ed81a11a47fb381b88192c7f300^{commit} # timeout=10
Checking out Revision d03b49f46de81ed81a11a47fb381b88192c7f300 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d03b49f46de81ed81a11a47fb381b88192c7f300 # timeout=10
Commit message: "Fix signature indentation of `SoftmaxSampling.transform()`"
 > git rev-list --no-walk 6e39c19747a40910cf4c3e2b4bfc8ba8ce824f93 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins14980577434502880029.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+23.gd03b49f.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.83,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.82,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.4.4,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+23.gd03b49f,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.2,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1041731016'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:26: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_ensemble.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_ensemble.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 128    112    12%

merlin/systems/dag/node.py                      20     20     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 66     66     0%

merlin/systems/dag/ops/feast.py                124    124     0%

merlin/systems/dag/ops/fil.py                  218    211     3%

merlin/systems/dag/ops/implicit.py              55     55     0%

merlin/systems/dag/ops/operator.py              70     60    14%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           131    121     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2257   1974    13%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:35: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 5.14s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins13571608617230567549.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 3, 2022

Click to view CI Results
GitHub pull request #204 of commit 5188742d4fcff71ef13ac2733c508303bb7c9e62, no merge conflicts.
Running as SYSTEM
Setting status of 5188742d4fcff71ef13ac2733c508303bb7c9e62 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/405/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 5188742d4fcff71ef13ac2733c508303bb7c9e62^{commit} # timeout=10
Checking out Revision 5188742d4fcff71ef13ac2733c508303bb7c9e62 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5188742d4fcff71ef13ac2733c508303bb7c9e62 # timeout=10
Commit message: "Fix `exportable_backends` method on `InferenceOperator`"
 > git rev-list --no-walk 3463e587c1c98526f10cdd4daa8c4168dc7e722a # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins7235279404683529486.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+33.g5188742.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.85,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.84,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+33.g5188742,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='581073379'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                124    124     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              107    107     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           134    124     7%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2305   2022    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:35: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 5.25s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins15338274563090295582.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 3, 2022

Click to view CI Results
GitHub pull request #204 of commit 8fea2111c8097dce17b265ff53845c187913e7ba, no merge conflicts.
Running as SYSTEM
Setting status of 8fea2111c8097dce17b265ff53845c187913e7ba to PENDING with url https://10.20.13.93:8080/job/merlin_systems/406/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 8fea2111c8097dce17b265ff53845c187913e7ba^{commit} # timeout=10
Checking out Revision 8fea2111c8097dce17b265ff53845c187913e7ba (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8fea2111c8097dce17b265ff53845c187913e7ba # timeout=10
Commit message: "Remove Python export from `PredictPyTorch` op"
 > git rev-list --no-walk 5188742d4fcff71ef13ac2733c508303bb7c9e62 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins607612408838368429.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+36.g8fea211.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.85,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.84,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+36.g8fea211,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2534895344'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:35: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 4.97s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins13273168255872007508.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 3, 2022

Click to view CI Results
GitHub pull request #204 of commit 6f37e696c4b9af38402207dd955ff96739381879, no merge conflicts.
Running as SYSTEM
Setting status of 6f37e696c4b9af38402207dd955ff96739381879 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/407/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 6f37e696c4b9af38402207dd955ff96739381879^{commit} # timeout=10
Checking out Revision 6f37e696c4b9af38402207dd955ff96739381879 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6f37e696c4b9af38402207dd955ff96739381879 # timeout=10
Commit message: "Minor fixes to FAISS tests"
 > git rev-list --no-walk 8fea2111c8097dce17b265ff53845c187913e7ba # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins10197372839541821353.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+39.g6f37e69.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.85,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.84,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+39.g6f37e69,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='4103444040'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:35: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 5.08s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins13731518777555508305.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit 324ec023e5819f919514e4b1bf81a1f1486c2e07, no merge conflicts.
Running as SYSTEM
Setting status of 324ec023e5819f919514e4b1bf81a1f1486c2e07 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/408/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 324ec023e5819f919514e4b1bf81a1f1486c2e07^{commit} # timeout=10
Checking out Revision 324ec023e5819f919514e4b1bf81a1f1486c2e07 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 324ec023e5819f919514e4b1bf81a1f1486c2e07 # timeout=10
Commit message: "Remove stray scratch file"
 > git rev-list --no-walk 6f37e696c4b9af38402207dd955ff96739381879 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins5906002923425013588.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+40.g324ec02.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+40.g324ec02,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3304783493'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:35: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 4.99s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins11098889568856156469.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit e98bc2e3409975ca27e0539ea135f82639c6bf76, no merge conflicts.
Running as SYSTEM
Setting status of e98bc2e3409975ca27e0539ea135f82639c6bf76 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/409/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse e98bc2e3409975ca27e0539ea135f82639c6bf76^{commit} # timeout=10
Checking out Revision e98bc2e3409975ca27e0539ea135f82639c6bf76 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e98bc2e3409975ca27e0539ea135f82639c6bf76 # timeout=10
Commit message: "Fix linter errors"
 > git rev-list --no-walk 324ec023e5819f919514e4b1bf81a1f1486c2e07 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins13547496421501968469.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+41.ge98bc2e.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+41.ge98bc2e,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3580357679'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:34: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 4.98s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins12524577812126796378.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit ebca19a47c526884d8eb826f5ff4ceffed050cd0, no merge conflicts.
Running as SYSTEM
Setting status of ebca19a47c526884d8eb826f5ff4ceffed050cd0 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/410/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse ebca19a47c526884d8eb826f5ff4ceffed050cd0^{commit} # timeout=10
Checking out Revision ebca19a47c526884d8eb826f5ff4ceffed050cd0 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ebca19a47c526884d8eb826f5ff4ceffed050cd0 # timeout=10
Commit message: "Fix dev requirements (`click` listed twice)"
 > git rev-list --no-walk e98bc2e3409975ca27e0539ea135f82639c6bf76 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins15617372719474659066.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+42.gebca19a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+42.gebca19a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='327525692'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:34: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 4.99s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins5138191389668159434.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit e4ec57acc1399368f185d435f6d92dc40a1e9090, no merge conflicts.
Running as SYSTEM
Setting status of e4ec57acc1399368f185d435f6d92dc40a1e9090 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/411/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse e4ec57acc1399368f185d435f6d92dc40a1e9090^{commit} # timeout=10
Checking out Revision e4ec57acc1399368f185d435f6d92dc40a1e9090 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e4ec57acc1399368f185d435f6d92dc40a1e9090 # timeout=10
Commit message: "Adjust `pylint.rc` to avoid false positives"
 > git rev-list --no-walk ebca19a47c526884d8eb826f5ff4ceffed050cd0 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins13613156853494368322.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.5.0+43.ge4ec57a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+43.ge4ec57a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1316767361'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:34: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 4.93s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins16301896380504401930.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit b43df27473cf95545a029793b2d165050a526a5b, no merge conflicts.
Running as SYSTEM
Setting status of b43df27473cf95545a029793b2d165050a526a5b to PENDING with url https://10.20.13.93:8080/job/merlin_systems/412/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse b43df27473cf95545a029793b2d165050a526a5b^{commit} # timeout=10
Checking out Revision b43df27473cf95545a029793b2d165050a526a5b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b43df27473cf95545a029793b2d165050a526a5b # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk e4ec57acc1399368f185d435f6d92dc40a1e9090 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins14506442049842281353.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+40.gb43df27.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+40.gb43df27,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3242344952'
test-gpu run-test: commands[0] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 6 items / 12 errors / 5 skipped

==================================== ERRORS ====================================

___________ ERROR collecting tests/unit/systems/dag/test_ensemble.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_ensemble.py:18: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

__________ ERROR collecting tests/unit/systems/dag/test_executors.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_executors.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_executors.py:22: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

________ ERROR collecting tests/unit/systems/dag/test_model_registry.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_model_registry.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_model_registry.py:25: in 

from merlin.systems.dag.ops.operator import InferenceOperator  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/dag/test_op_runner.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/test_op_runner.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/test_op_runner.py:27: in 

from merlin.dag import DictArray, Graph

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/dag/ops/test_ops.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/dag/ops/test_ops.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/dag/ops/test_ops.py:23: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

________ ERROR collecting tests/unit/systems/ops/faiss/test_executor.py ________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/faiss/test_executor.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/faiss/test_executor.py:19: in 

from merlin.dag import DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

___________ ERROR collecting tests/unit/systems/ops/feast/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/feast/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/feast/test_op.py:8: in 

from merlin.dag import ColumnSelector, DictArray

E   ImportError: cannot import name 'DictArray' from 'merlin.dag' (/var/jenkins_home/.local/lib/python3.8/site-packages/merlin/dag/init.py)

_________ ERROR collecting tests/unit/systems/ops/fil/test_ensemble.py _________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_ensemble.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_ensemble.py:30: in 

from merlin.systems.dag.ops.fil import PredictForest  # noqa

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

__________ ERROR collecting tests/unit/systems/ops/fil/test_forest.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_forest.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_forest.py:30: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

____________ ERROR collecting tests/unit/systems/ops/fil/test_op.py ____________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/fil/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/fil/test_op.py:26: in 

from merlin.systems.dag.ops import fil as fil_op

merlin/systems/dag/ops/fil.py:24: in 

from merlin.core.protocols import Transformable

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

_________ ERROR collecting tests/unit/systems/ops/implicit/test_op.py __________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/implicit/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/implicit/test_op.py:26: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

___________ ERROR collecting tests/unit/systems/ops/torch/test_op.py ___________

ImportError while importing test module '/var/jenkins_home/workspace/merlin_systems/systems/tests/unit/systems/ops/torch/test_op.py'.

Hint: make sure your test modules/packages have valid Python names.

Traceback:

/usr/lib/python3.8/importlib/init.py:127: in import_module

return _bootstrap._gcd_import(name[level:], package, level)

tests/unit/systems/ops/torch/test_op.py:29: in 

from merlin.systems.dag.ensemble import Ensemble

merlin/systems/dag/init.py:19: in 

from .ensemble import Ensemble

merlin/systems/dag/ensemble.py:37: in 

from merlin.systems.dag.ops.operator import add_model_param  # noqa

merlin/systems/dag/ops/operator.py:15: in 

from merlin.core.protocols import Transformable  # noqa

E   ModuleNotFoundError: No module named 'merlin.core.protocols'

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 34 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      2    33%

merlin/systems/dag/ensemble.py                 151    135    11%

merlin/systems/dag/node.py                      23     23     0%

merlin/systems/dag/op_runner.py                 26     26     0%

merlin/systems/dag/ops/init.py               9      8    11%

merlin/systems/dag/ops/compat.py                24     24     0%

merlin/systems/dag/ops/faiss.py                 71     71     0%

merlin/systems/dag/ops/feast.py                126    126     0%

merlin/systems/dag/ops/fil.py                  221    214     3%

merlin/systems/dag/ops/implicit.py              58     58     0%

merlin/systems/dag/ops/operator.py              75     65    13%

merlin/systems/dag/ops/pytorch.py              104    104     0%

merlin/systems/dag/ops/session_filter.py        44     44     0%

merlin/systems/dag/ops/softmax_sampling.py      51     51     0%

merlin/systems/dag/ops/tensorflow.py           129    119     8%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     72     0%

merlin/systems/model_registry.py                16      8    50%

merlin/systems/triton/init.py               47     33    30%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    239    10%

merlin/systems/triton/utils.py                  72     54    25%

merlin/systems/workflow/init.py             22     20     9%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299   2016    12%


=========================== short test summary info ============================

SKIPPED [1] tests/unit/systems/dag/test_graph.py:25: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_ensemble.py:34: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/nvtabular/test_op.py:31: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_ensemble.py:38: could not import 'merlin.systems.dag.ensemble': No module named 'merlin.core.protocols'

SKIPPED [1] tests/unit/systems/ops/tf/test_op.py:33: could not import 'merlin.systems.dag.ops.tensorflow': No module named 'merlin.core.protocols'

!!!!!!!!!!!!!!!!!!! Interrupted: 12 errors during collection !!!!!!!!!!!!!!!!!!!

================== 5 skipped, 36 warnings, 12 errors in 4.98s ==================

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 2)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins13672455005104607368.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit a5b683f6d0e0763a2ece9e22be1fbc031ddaa6ee, no merge conflicts.
Running as SYSTEM
Setting status of a5b683f6d0e0763a2ece9e22be1fbc031ddaa6ee to PENDING with url https://10.20.13.93:8080/job/merlin_systems/413/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse a5b683f6d0e0763a2ece9e22be1fbc031ddaa6ee^{commit} # timeout=10
Checking out Revision a5b683f6d0e0763a2ece9e22be1fbc031ddaa6ee (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a5b683f6d0e0763a2ece9e22be1fbc031ddaa6ee # timeout=10
Commit message: "Make tests run against the latest version of Core from `main`"
 > git rev-list --no-walk b43df27473cf95545a029793b2d165050a526a5b # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins7075448317678875391.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+41.ga5b683f.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+41.ga5b683f,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1519178821'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-o3fjtppw
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-o3fjtppw
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+3.ge3286f8-py3-none-any.whl size=117553 sha256=c0bc0336c1099d9e505b0cc628d92a06380d85de862a1b9f86d6e3e19a08d2b5
  Stored in directory: /tmp/pip-ephem-wheel-cache-tykmp9n7/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+3.ge3286f8
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py F [  3%]

F                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_____________________ test_example_serving_implicit[True] ______________________


tb = <testbook.client.TestbookNotebookClient object at 0x7f93e838c2b0>

gpu = True


@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
@testbook(REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_implicit(tb, gpu):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """,
        pop=True,
    )
    NUM_OF_CELLS = len(tb.cells)
    if not gpu:
        tb.cells[
            6
        ].source = (
            "model = BayesianPersonalizedRanking(use_gpu=False)\nmodel.fit(train_transformed)"
        )
    tb.execute_cell(list(range(0, 16)))





  with run_triton_server("ensemble", grpc_port=8001):





tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:54:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = 'ensemble'


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode
                            raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")

                        try:
                            ready = client.is_server_ready()
                        except tritonclient.utils.InferenceServerException:
                            ready = False

                        if ready:
                            yield client
                            return

                        time.sleep(1)
                    LOG.error("Failed to start tritonserver in %s seconds", seconds)





              raise RuntimeError("Timed out waiting for tritonserver to become ready")





E                   RuntimeError: Timed out waiting for tritonserver to become ready


merlin/systems/triton/utils.py:95: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 19:12:55.947521 4529 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f7de6000000' with size 268435456

I1004 19:12:55.948288 4529 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1004 19:12:55.954166 4529 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 19:12:55.954213 4529 model_lifecycle.cc:459] loading: 1_predictforest:1

I1004 19:12:55.954253 4529 model_lifecycle.cc:459] loading: 1_fil:1

I1004 19:12:55.954300 4529 model_lifecycle.cc:459] loading: 1_predictimplicit:1

I1004 19:13:00.338967 4529 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1004 19:13:00.339004 4529 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1004 19:13:00.339011 4529 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1004 19:13:00.341125 4529 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 19:13:03.274194 4529 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

1004 19:13:06.041789 4690 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


1004 19:20:56.744579 4612 pb_stub.cc:1016] Non-graceful termination detected.

------------------------------ Captured log call -------------------------------

ERROR    merlin-systems:utils.py:93 Failed to start tritonserver in 60 seconds

ERROR    merlin-systems:utils.py:93 Failed to start tritonserver in 120 seconds

ERROR    merlin-systems:utils.py:93 Failed to start tritonserver in 300 seconds

_____________________ test_example_serving_implicit[False] _____________________


tb = <testbook.client.TestbookNotebookClient object at 0x7f93e838c2b0>

gpu = False


@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
@testbook(REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_implicit(tb, gpu):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """,
        pop=True,
    )
    NUM_OF_CELLS = len(tb.cells)
    if not gpu:
        tb.cells[
            6
        ].source = (
            "model = BayesianPersonalizedRanking(use_gpu=False)\nmodel.fit(train_transformed)"
        )
    tb.execute_cell(list(range(0, 16)))





  with run_triton_server("ensemble", grpc_port=8001):





tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:54:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = 'ensemble'


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 19:21:04.712323 4844 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f7244000000' with size 268435456

I1004 19:21:04.713062 4844 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1004 19:21:04.718385 4844 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 19:21:04.718423 4844 model_lifecycle.cc:459] loading: 1_predictforest:1

I1004 19:21:04.718450 4844 model_lifecycle.cc:459] loading: 1_fil:1

I1004 19:21:04.718476 4844 model_lifecycle.cc:459] loading: 1_predictimplicit:1

I1004 19:21:04.737918 4844 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1004 19:21:04.737943 4844 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1004 19:21:04.737949 4844 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1004 19:21:04.739253 4844 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_fil (version 1)

I1004 19:21:04.744955 4844 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_fil_0 (GPU device 0)

I1004 19:21:04.799991 4844 model_lifecycle.cc:693] successfully loaded '1_fil' version 1

1004 19:21:11.880992 4939 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1004 19:21:11.936759 4844 model_lifecycle.cc:596] failed to load '1_predictforest' version 1: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:21:15.984455 4844 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 19:21:18.906880 4844 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 1_predictimplicit (GPU device 0)

I1004 19:21:18.907200 4844 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 19:21:21.712252 4844 model_lifecycle.cc:693] successfully loaded '1_predictimplicit' version 1

I1004 19:21:21.712601 4844 model_lifecycle.cc:459] loading: ensemble_model:1

I1004 19:21:21.712954 4844 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1004 19:21:21.713079 4844 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 19:21:21.713173 4844 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| fil     | /opt/tritonserver/backends/fil/libtriton_fil.so       | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:21:21.713387 4844 server.cc:631]

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model               | Version | Status                                                                                                                                                                                                                                               |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_transformworkflow | 1       | READY                                                                                                                                                                                                                                                |

| 1_fil               | 1       | READY                                                                                                                                                                                                                                                |

| 1_predictforest     | 1       | UNAVAILABLE: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py) |

|                     |         |                                                                                                                                                                                                                                                      |

|                     |         | At:                                                                                                                                                                                                                                                  |

|                     |         |   /var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35):                                                                                                                                                |

|                     |         |   (219): _call_with_frames_removed                                                                                                                                                                                      |

|                     |         |   (848): exec_module                                                                                                                                                                                           |

|                     |         |   (686): _load_unlocked                                                                                                                                                                                                 |

|                     |         |   (975): _find_and_load_unlocked                                                                                                                                                                                        |

|                     |         |   (991): _find_and_load                                                                                                                                                                                                 |

| 1_predictimplicit   | 1       | READY                                                                                                                                                                                                                                                |

| ensemble_model      | 1       | READY                                                                                                                                                                                                                                                |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:21:21.776564 4844 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 19:21:21.777413 4844 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | ensemble                                                                                                                                                                                     |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:21:21.777449 4844 server.cc:262] Waiting for in-flight requests to complete.

I1004 19:21:21.777461 4844 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 19:21:21.777680 4844 server.cc:293] All models are stopped, unloading models

I1004 19:21:21.777692 4844 server.cc:300] Timeout 30: Found 4 live models and 0 in-flight non-inference requests

I1004 19:21:21.777786 4844 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1004 19:21:21.777837 4844 instance_finalize.hpp:36] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1004 19:21:21.778082 4844 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state

I1004 19:21:21.778146 4844 model_lifecycle.cc:578] successfully unloaded '1_fil' version 1

I1004 19:21:22.777782 4844 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 19:21:22.800892 4844 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:21:23.284545 4844 model_lifecycle.cc:578] successfully unloaded '1_predictimplicit' version 1

I1004 19:21:23.389853 4844 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 19:21:23.777917 4844 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 19:21:23.801079 4844 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-10/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7f93441eae50>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-10/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 19:26:18.374379 8556 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f0064000000' with size 268435456

I1004 19:26:18.375206 8556 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 19:26:18.379342 8556 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 19:26:18.379401 8556 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 19:26:18.379437 8556 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:26:27.121390 8556 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 19:26:30.102944 8556 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 19:26:30.103171 8556 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 19:26:33.594889 8556 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 19:26:33.595082 8556 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 19:26:33.595198 8556 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:26:33.595276 8556 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 19:26:33.658878 8556 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 19:26:33.659737 8556 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-10/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:26:33.659775 8556 server.cc:262] Waiting for in-flight requests to complete.

I1004 19:26:33.659783 8556 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 19:26:33.659842 8556 server.cc:293] All models are stopped, unloading models

I1004 19:26:33.659852 8556 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 19:26:34.659933 8556 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 19:26:34.683095 8556 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:26:35.138322 8556 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 19:26:35.165571 8556 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 19:26:35.660057 8556 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 19:26:35.683294 8556 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py              104     21    80%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           129     27    79%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     10    86%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299    891    61%


============ 3 failed, 80 passed, 56 warnings in 985.89s (0:16:25) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins14140513330375672311.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit fe59784882cd0511f64c83569ca4800669872fd1, no merge conflicts.
Running as SYSTEM
Setting status of fe59784882cd0511f64c83569ca4800669872fd1 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/414/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse fe59784882cd0511f64c83569ca4800669872fd1^{commit} # timeout=10
Checking out Revision fe59784882cd0511f64c83569ca4800669872fd1 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fe59784882cd0511f64c83569ca4800669872fd1 # timeout=10
Commit message: "Make tests run against the latest version of Core from `main`"
 > git rev-list --no-walk a5b683f6d0e0763a2ece9e22be1fbc031ddaa6ee # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins13792122076824580696.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+41.gfe59784.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+41.gfe59784,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2314590047'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-i_9oxcea
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-i_9oxcea
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-11/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7fa2d4434190>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-11/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 19:35:46.009132 15239 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f5826000000' with size 268435456

I1004 19:35:46.009896 15239 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 19:35:46.013965 15239 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 19:35:46.014026 15239 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 19:35:46.014063 15239 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:35:54.735017 15239 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 19:35:57.660848 15239 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 19:35:57.662007 15239 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 19:36:01.142063 15239 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 19:36:01.142222 15239 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 19:36:01.142336 15239 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:36:01.142422 15239 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 19:36:01.205326 15239 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 19:36:01.206173 15239 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-11/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:36:01.206213 15239 server.cc:262] Waiting for in-flight requests to complete.

I1004 19:36:01.206223 15239 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 19:36:01.206279 15239 server.cc:293] All models are stopped, unloading models

I1004 19:36:01.206290 15239 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 19:36:02.206373 15239 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 19:36:02.233087 15239 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:36:02.743036 15239 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 19:36:02.771381 15239 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 19:36:03.206511 15239 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 19:36:03.233296 15239 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py              104     21    80%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           129     27    79%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2299    893    61%


============ 1 failed, 82 passed, 56 warnings in 529.87s (0:08:49) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins7775582481161224861.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit 99198ae0d5962986b9337d9c9722f3b25cb294af, no merge conflicts.
Running as SYSTEM
Setting status of 99198ae0d5962986b9337d9c9722f3b25cb294af to PENDING with url https://10.20.13.93:8080/job/merlin_systems/415/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 99198ae0d5962986b9337d9c9722f3b25cb294af^{commit} # timeout=10
Checking out Revision 99198ae0d5962986b9337d9c9722f3b25cb294af (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 99198ae0d5962986b9337d9c9722f3b25cb294af # timeout=10
Commit message: "Remove unnecessary `from_config` methods from TF/Torch ops"
 > git rev-list --no-walk fe59784882cd0511f64c83569ca4800669872fd1 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins9126980099179931560.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+43.g99198ae.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+43.g99198ae,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='444066652'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-gum_80ko
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-gum_80ko
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-12/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7fc7d04282e0>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-12/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 19:45:23.032792 21959 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fa8a4000000' with size 268435456

I1004 19:45:23.033547 21959 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 19:45:23.037601 21959 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 19:45:23.037663 21959 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 19:45:23.037701 21959 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:45:31.772507 21959 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 19:45:34.727942 21959 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 19:45:34.728263 21959 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 19:45:38.130935 21959 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 19:45:38.131146 21959 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 19:45:38.131266 21959 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:45:38.131342 21959 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 19:45:38.195589 21959 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 19:45:38.196454 21959 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-12/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:45:38.196493 21959 server.cc:262] Waiting for in-flight requests to complete.

I1004 19:45:38.196504 21959 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 19:45:38.196594 21959 server.cc:293] All models are stopped, unloading models

I1004 19:45:38.196605 21959 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 19:45:39.196692 21959 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 19:45:39.225333 21959 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:45:39.790777 21959 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 19:45:39.817997 21959 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 19:45:40.196820 21959 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 19:45:40.225529 21959 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py               93     13    86%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           116     17    85%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2275    875    62%


============ 1 failed, 82 passed, 56 warnings in 535.14s (0:08:55) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins4379935493566065163.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit 9e396b51971361112eaa45cdd8a323f170fbaab6, no merge conflicts.
Running as SYSTEM
Setting status of 9e396b51971361112eaa45cdd8a323f170fbaab6 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/416/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 9e396b51971361112eaa45cdd8a323f170fbaab6^{commit} # timeout=10
Checking out Revision 9e396b51971361112eaa45cdd8a323f170fbaab6 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9e396b51971361112eaa45cdd8a323f170fbaab6 # timeout=10
Commit message: "Adjust executor tests for CPU compatibility"
 > git rev-list --no-walk 99198ae0d5962986b9337d9c9722f3b25cb294af # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins15185803531984042147.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+44.g9e396b5.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+44.g9e396b5,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='294540761'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-yr816427
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-yr816427
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py .EE.                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


==================================== ERRORS ====================================

_______ ERROR at setup of test_run_dag_on_dataframe_with_local_executor ________

Error evaluating 'skipif': you need to specify reason=STRING when using booleans as conditions.

________ ERROR at setup of test_run_dag_on_dataframe_with_dask_executor ________

Error evaluating 'skipif': you need to specify reason=STRING when using booleans as conditions.

=================================== FAILURES ===================================

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-13/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7fe8e46285e0>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-13/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 19:54:46.798452 28672 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f1a6e000000' with size 268435456

I1004 19:54:46.799197 28672 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 19:54:46.803298 28672 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 19:54:46.803360 28672 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 19:54:46.803397 28672 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:54:55.530909 28672 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 19:54:58.499289 28672 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 19:54:58.499595 28672 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 19:55:01.958921 28672 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 19:55:01.959054 28672 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 19:55:01.959141 28672 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:55:01.959194 28672 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 19:55:02.022107 28672 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 19:55:02.022952 28672 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-13/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 19:55:02.022992 28672 server.cc:262] Waiting for in-flight requests to complete.

I1004 19:55:02.023001 28672 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 19:55:02.023070 28672 server.cc:293] All models are stopped, unloading models

I1004 19:55:02.023080 28672 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 19:55:03.023164 28672 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 19:55:03.044616 28672 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 19:55:03.560500 28672 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 19:55:03.591536 28672 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 19:55:04.023297 28672 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 19:55:04.044817 28672 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py               93     13    86%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           116     17    85%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2275    875    62%


======= 1 failed, 80 passed, 56 warnings, 2 errors in 545.50s (0:09:05) ========

/usr/local/lib/python3.8/dist-packages/coverage/data.py:130: CoverageWarning: Couldn't use data file '/var/jenkins_home/workspace/merlin_systems/systems/.coverage.10.20.17.231.25990.744352': database disk image is malformed

data._warn(str(exc))

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins6453213097820864584.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit d1060046576b042747243074db8e23b41a54b8ea, no merge conflicts.
Running as SYSTEM
Setting status of d1060046576b042747243074db8e23b41a54b8ea to PENDING with url https://10.20.13.93:8080/job/merlin_systems/417/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse d1060046576b042747243074db8e23b41a54b8ea^{commit} # timeout=10
Checking out Revision d1060046576b042747243074db8e23b41a54b8ea (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d1060046576b042747243074db8e23b41a54b8ea # timeout=10
Commit message: "Remove TODO list"
 > git rev-list --no-walk 9e396b51971361112eaa45cdd8a323f170fbaab6 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins1147090820480881401.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+45.gd106004.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+45.gd106004,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='565375928'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-_4or68ej
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-_4or68ej
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py .EE.                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


==================================== ERRORS ====================================

_______ ERROR at setup of test_run_dag_on_dataframe_with_local_executor ________

Error evaluating 'skipif': you need to specify reason=STRING when using booleans as conditions.

________ ERROR at setup of test_run_dag_on_dataframe_with_dask_executor ________

Error evaluating 'skipif': you need to specify reason=STRING when using booleans as conditions.

=================================== FAILURES ===================================

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-14/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7f4818218070>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-14/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 20:04:34.021457 3000 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f24a6000000' with size 268435456

I1004 20:04:34.022237 3000 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 20:04:34.026310 3000 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 20:04:34.026371 3000 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 20:04:34.026408 3000 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:04:42.764641 3000 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 20:04:45.679992 3000 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 20:04:45.680303 3000 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 20:04:49.119077 3000 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 20:04:49.119240 3000 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 20:04:49.119356 3000 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:04:49.119433 3000 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 20:04:49.182762 3000 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 20:04:49.183616 3000 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-14/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:04:49.183654 3000 server.cc:262] Waiting for in-flight requests to complete.

I1004 20:04:49.183664 3000 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 20:04:49.183755 3000 server.cc:293] All models are stopped, unloading models

I1004 20:04:49.183767 3000 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 20:04:50.183849 3000 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 20:04:50.204291 3000 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:04:50.747956 3000 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 20:04:50.776932 3000 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 20:04:51.183979 3000 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 20:04:51.204464 3000 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py               93     13    86%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           116     17    85%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2275    875    62%


======= 1 failed, 80 passed, 56 warnings, 2 errors in 538.10s (0:08:58) ========

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins13790602497901469991.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit 9b4db4bc421c4f345478eb24d12405b9fbbf670b, no merge conflicts.
Running as SYSTEM
Setting status of 9b4db4bc421c4f345478eb24d12405b9fbbf670b to PENDING with url https://10.20.13.93:8080/job/merlin_systems/418/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 9b4db4bc421c4f345478eb24d12405b9fbbf670b^{commit} # timeout=10
Checking out Revision 9b4db4bc421c4f345478eb24d12405b9fbbf670b (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 9b4db4bc421c4f345478eb24d12405b9fbbf670b # timeout=10
Commit message: "Add a reason for skipif in executor tests"
 > git rev-list --no-walk d1060046576b042747243074db8e23b41a54b8ea # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins6305897308912631120.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+46.g9b4db4b.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+46.g9b4db4b,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1131135626'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-wtr_hw1j
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-wtr_hw1j
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py .F                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

______________________ test_filter_candidates_with_triton ______________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-15/test_filter_candidates_with_tr0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_filter_candidates_with_triton(tmpdir):
    request_schema = Schema(
        [
            ColumnSchema("candidate_ids", dtype=np.int32),
            ColumnSchema("movie_ids", dtype=np.int32),
        ]
    )

    candidate_ids = np.random.randint(1, 100000, 100).astype(np.int32)
    movie_ids_1 = np.zeros(100, dtype=np.int32)
    movie_ids_1[:20] = np.unique(candidate_ids)[:20]

    combined_features = {
        "candidate_ids": candidate_ids,
        "movie_ids": movie_ids_1,
    }

    request_df = make_df(combined_features)

    filtering = ["candidate_ids"] >> FilterCandidates(filter_out=["movie_ids"])

    ensemble = Ensemble(filtering, request_schema)
    ens_config, node_configs = ensemble.export(tmpdir)

    response = run_ensemble_on_tritonserver(
        tmpdir, request_schema, request_df, ensemble.output_schema.column_names, "ensemble_model"
    )

    assert response is not None





  assert len(response["filtered_ids"]) == 80





E       assert 79 == 80

E        +  where 79 = len(array([47262, 56432, 63353, 99303, 50239, 78147, 75939, 58647, 36623,\n       63309, 81777, 41532, 75235, 77086, 22103,... 66426, 56762, 80112, 98700, 76637, 79832, 76158,\n       47608, 53293, 49684, 36748, 31010, 91669, 99967], dtype=int32))


tests/unit/systems/dag/ops/test_ops.py:89: AssertionError

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------

I1004 20:11:54.724079 8257 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7ff196000000' with size 268435456

I1004 20:11:54.724817 8257 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1004 20:11:54.728037 8257 model_lifecycle.cc:459] loading: 0_filtercandidates:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:11:58.767858 8257 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_filtercandidates (GPU device 0)

I1004 20:12:01.514702 8257 model_lifecycle.cc:693] successfully loaded '0_filtercandidates' version 1

I1004 20:12:01.515008 8257 model_lifecycle.cc:459] loading: ensemble_model:1

I1004 20:12:01.515186 8257 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1004 20:12:01.515273 8257 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 20:12:01.515362 8257 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:12:01.515428 8257 server.cc:631]

+--------------------+---------+--------+

| Model              | Version | Status |

+--------------------+---------+--------+

| 0_filtercandidates | 1       | READY  |

| ensemble_model     | 1       | READY  |

+--------------------+---------+--------+


I1004 20:12:01.579326 8257 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 20:12:01.580161 8257 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-15/test_filter_candidates_with_tr0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:12:01.581651 8257 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1004 20:12:01.582156 8257 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1004 20:12:01.623327 8257 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1004 20:12:02.601522 8257 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1004 20:12:03.601728 8257 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1004 20:12:04.542296 8257 server.cc:262] Waiting for in-flight requests to complete.

I1004 20:12:04.542329 8257 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 20:12:04.542519 8257 server.cc:293] All models are stopped, unloading models

I1004 20:12:04.542542 8257 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 20:12:04.542581 8257 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

W1004 20:12:04.620775 8257 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1004 20:12:05.542638 8257 server.cc:300] Timeout 29: Found 1 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:12:06.113410 8257 model_lifecycle.cc:578] successfully unloaded '0_filtercandidates' version 1

I1004 20:12:06.542774 8257 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-15/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7f3e846a3310>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-15/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 20:13:54.909524 9691 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f5bd6000000' with size 268435456

I1004 20:13:54.910293 9691 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 20:13:54.914307 9691 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 20:13:54.914369 9691 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 20:13:54.914404 9691 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:14:03.619195 9691 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 20:14:06.601119 9691 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 20:14:06.601424 9691 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 20:14:10.036176 9691 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 20:14:10.036327 9691 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 20:14:10.036440 9691 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:14:10.036522 9691 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 20:14:10.099510 9691 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 20:14:10.100373 9691 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-15/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:14:10.100413 9691 server.cc:262] Waiting for in-flight requests to complete.

I1004 20:14:10.100424 9691 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 20:14:10.100484 9691 server.cc:293] All models are stopped, unloading models

I1004 20:14:10.100494 9691 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 20:14:11.100581 9691 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 20:14:11.114339 9691 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:14:11.623076 9691 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 20:14:11.650155 9691 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 20:14:12.100712 9691 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 20:14:12.114532 9691 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py               93     13    86%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           116     17    85%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2275    875    62%


============ 2 failed, 81 passed, 56 warnings in 528.77s (0:08:48) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins17550656107710204527.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 4, 2022

Click to view CI Results
GitHub pull request #204 of commit 67d18bf773e5b227788ff0df7039ffbe19342174, no merge conflicts.
Running as SYSTEM
Setting status of 67d18bf773e5b227788ff0df7039ffbe19342174 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/419/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 67d18bf773e5b227788ff0df7039ffbe19342174^{commit} # timeout=10
Checking out Revision 67d18bf773e5b227788ff0df7039ffbe19342174 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67d18bf773e5b227788ff0df7039ffbe19342174 # timeout=10
Commit message: "Remove `_python` property from TF/Torch ops"
 > git rev-list --no-walk 9b4db4bc421c4f345478eb24d12405b9fbbf670b # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins7746992714348562247.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu inst-nodeps: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+47.g67d18bf.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.86,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.85,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+47.g67d18bf,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='2234772105'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-udpcij89
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-udpcij89
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit e3286f8f62657cf17ee9fecab3b23f01564f5d45
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (21.3)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+3.ge3286f8) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+3.ge3286f8) (0.55.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.2.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.4)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.0.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+3.ge3286f8) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+3.ge3286f8) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+3.ge3286f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+3.ge3286f8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+3.ge3286f8) (4.0.0)
test-gpu run-test: commands[1] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

___________________ test_workflow_op_serving_python[parquet] ___________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-16/test_workflow_op_serving_pytho0')

dataset = <merlin.io.dataset.Dataset object at 0x7fe88c03b310>

engine = 'parquet'


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("engine", ["parquet"])
def test_workflow_op_serving_python(tmpdir, dataset, engine):
    input_columns = ["x", "y", "id"]

    # NVT
    workflow_ops = input_columns >> wf_ops.Rename(postfix="_nvt")
    workflow = Workflow(workflow_ops)
    workflow.fit(dataset)

    # Triton
    triton_op = "*" >> workflow_op.TransformWorkflow(
        workflow,
        conts=["x_nvt", "y_nvt"],
        cats=["id_nvt"],
        backend="python",
    )

    wkflow_ensemble = ensemble.Ensemble(triton_op, workflow.input_schema)
    ens_config, node_configs = wkflow_ensemble.export(tmpdir)

    input_data = {}
    inputs = []
    for col_name, col_schema in workflow.input_schema.column_schemas.items():
        col_dtype = col_schema.dtype
        input_data[col_name] = np.array([[2], [3], [4]]).astype(col_dtype)

        triton_input = grpcclient.InferInput(
            col_name, input_data[col_name].shape, triton.np_to_triton_dtype(col_dtype)
        )

        triton_input.set_data_from_numpy(input_data[col_name])

        inputs.append(triton_input)

    outputs = []
    for col_name in workflow.output_schema.column_names:
        outputs.append(grpcclient.InferRequestedOutput(col_name))

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/nvtabular/test_ensemble.py:78:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-16/test_workflow_op_serving_pytho0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1004 20:23:11.902591 16429 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f4e66000000' with size 268435456

I1004 20:23:11.903392 16429 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

E1004 20:23:11.907549 16429 model_repository_manager.cc:996] Poll failed for model directory 'ensemble_model': must specify 'input_map' in step 0 of ensemble 'ensemble_model'

I1004 20:23:11.907630 16429 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1004 20:23:11.907685 16429 model_lifecycle.cc:459] loading: 0_py_transformworkflow:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:23:20.638338 16429 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_py_transformworkflow (GPU device 0)

I1004 20:23:24.112319 16429 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1004 20:23:24.112614 16429 model_lifecycle.cc:693] successfully loaded '0_py_transformworkflow' version 1

I1004 20:23:27.087304 16429 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1004 20:23:27.087463 16429 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1004 20:23:27.087575 16429 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:23:27.087654 16429 server.cc:631]

+------------------------+---------+--------+

| Model                  | Version | Status |

+------------------------+---------+--------+

| 0_py_transformworkflow | 1       | READY  |

| 0_transformworkflow    | 1       | READY  |

+------------------------+---------+--------+


I1004 20:23:27.152291 16429 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1004 20:23:27.153141 16429 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-16/test_workflow_op_serving_pytho0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1004 20:23:27.153178 16429 server.cc:262] Waiting for in-flight requests to complete.

I1004 20:23:27.153190 16429 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1004 20:23:27.153261 16429 server.cc:293] All models are stopped, unloading models

I1004 20:23:27.153272 16429 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1004 20:23:28.153354 16429 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1004 20:23:28.172997 16429 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1004 20:23:28.705370 16429 model_lifecycle.cc:578] successfully unloaded '0_py_transformworkflow' version 1

I1004 20:23:28.803661 16429 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1004 20:23:29.153480 16429 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1004 20:23:29.173202 16429 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 71     18    75%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              75      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        44      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     22    69%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2267    873    61%


============ 1 failed, 82 passed, 56 warnings in 530.21s (0:08:50) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins13098407030372910125.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 7, 2022

Click to view CI Results
GitHub pull request #204 of commit 54b6849bd9436c0f6fb2fe82937ed1820d40de34, no merge conflicts.
Running as SYSTEM
Setting status of 54b6849bd9436c0f6fb2fe82937ed1820d40de34 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/432/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 54b6849bd9436c0f6fb2fe82937ed1820d40de34^{commit} # timeout=10
Checking out Revision 54b6849bd9436c0f6fb2fe82937ed1820d40de34 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 54b6849bd9436c0f6fb2fe82937ed1820d40de34 # timeout=10
Commit message: "Skip the Triton executor model test when Triton isn't available"
 > git rev-list --no-walk 93dc6e0f178454feb813a4abb62021f020e3a46b # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins12410389693272398927.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+9.g54b6849.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.89,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.88,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+9.g54b6849,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3332552349'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-k8aodreq
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-k8aodreq
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 14a18dc0de5d5fd7737ecbadf9f6d7fa5d801b67
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+9.g14a18dc) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+9.g14a18dc-py3-none-any.whl size=118253 sha256=cba6cf713d28da27c39d9f498f7b54b87b50a7a6b2695f1ec5b608cad0c06b14
  Stored in directory: /tmp/pip-ephem-wheel-cache-kdw4krmr/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+9.g14a18dc
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+9.g54b6849) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+9.g54b6849) (0.7.0+9.g14a18dc)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.3.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (0.55.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (4.64.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (5.8.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (3.1.2)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (8.1.3)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.7.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+9.g54b6849) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.6.0+9.g54b6849-py3-none-any.whl size=83387 sha256=ac6b11c170dcc9d39728908edc7926f6aecbe2a4fdfe6db3de7afe3d84fda4b6
  Stored in directory: /tmp/pip-ephem-wheel-cache-xqxdz3aa/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.6.0+9.g54b6849
    Uninstalling merlin-systems-0.6.0+9.g54b6849:
      Successfully uninstalled merlin-systems-0.6.0+9.g54b6849
Successfully installed merlin-systems-0.6.0+9.g54b6849
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     14    81%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2272    881    61%


================= 83 passed, 56 warnings in 528.39s (0:08:48) ==================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins1965681938203673498.sh

@github-actions
Copy link

github-actions bot commented Oct 7, 2022

Documentation preview

https://nvidia-merlin.github.io/systems/review/pr-204

oliverholworthy
oliverholworthy reviewed Oct 7, 2022
View reviewed changes
merlin/systems/dag/ops/fil.py
params: dict = None,
node_id: int = None,
version: int = 1,
backend: str = "ensemble",
Copy link
Contributor

@oliverholworthy oliverholworthy Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This backend doesn't appear to be used in this method. I'm guessing it's here for consistency of export signatures?

Copy link
Contributor

@oliverholworthy oliverholworthy Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, There is one thing confusing me about this backend parameter. It's that the value of this backend for the PipelineableInferenceOperator export method means something different. In that method the backend corresponds to the backend that will be written out in the Triton config. Which suggest that the ensemble is only valid for the Ensemble and not for all operators.

Copy link
Contributor Author

@karlhigley karlhigley Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the backend corresponds to the backend that will be written out in the Triton config

This is always true, but there's a wobble in the logic here. An operator that isn't exportable for a given format (e.g. ensemble) won't have its export method called, but some operators need to write artifacts to disk in cases where they won't write out a Triton model config. In those cases, we currently consider them exportable for the ensemble back-end, but return an empty model config from the export method.

We could have resolved these weird inconsistencies by introducing back-ends/run-times/adapters in this PR (since then each back-end could have its own implementation of export for any given operator via the operator implementation table), but we wrote it this way in part to motivate the refactor in that direction.

Copy link
Contributor

@oliverholworthy oliverholworthy Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the context of this PredictForest operator, which currently has exportable_backends = ["ensemble", "executor"]. Based on what you described I would assume that for operators that include executor in exportable_backends then we would need to perform a conditional logic in the export method.

Copy link
Contributor

@oliverholworthy oliverholworthy Oct 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm understanding this correctly. I think what this means is that we'll end up having unused models written out to the model repository, but we'll also be exporting the node in the same way as for the ensemble case (if there's no conditional logic similar to the Faiss operator). I suppose that doesn't break any functionality (since, if the op contains executor in exportable_backends, then we'll be calling the transform method as part of the LocalExecutor), but maybe not the intention?

Copy link
Contributor Author

@karlhigley karlhigley Oct 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for operators that include executor in exportable_backends then we would need to perform a conditional logic in the export method

It depends on how the operator is structured. For operators that run in custom Triton back-ends, we still need to export a separate model directory. For operators that wrap one operator in another (e.g. in order to align the schema of the outer operator with the graph), the outer wrapper op runs in the Executor (in plain Python) but the inner op runs in a custom back-end.

Copy link
Contributor Author

@karlhigley karlhigley Oct 17, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose that doesn't break any functionality (since, if the op contains executor in exportable_backends, then we'll be calling the transform method as part of the LocalExecutor), but maybe not the intention?

I think this is a case where we're using the operator's ensemble export twice:

  • Once in the actual export of the full ensemble to run as a Triton ensemble
  • Once for exporting the single operator as a separate model directory to be used in Executor mode (still required to use specific non-Python Triton back-ends)

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 7, 2022

Click to view CI Results
GitHub pull request #204 of commit b5e3995782d5fcb0e15bbd2774eca36ed441f64d, no merge conflicts.
Running as SYSTEM
Setting status of b5e3995782d5fcb0e15bbd2774eca36ed441f64d to PENDING with url https://10.20.13.93:8080/job/merlin_systems/433/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse b5e3995782d5fcb0e15bbd2774eca36ed441f64d^{commit} # timeout=10
Checking out Revision b5e3995782d5fcb0e15bbd2774eca36ed441f64d (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b5e3995782d5fcb0e15bbd2774eca36ed441f64d # timeout=10
Commit message: "Use the `_parse_model_repository` fn in executor model"
 > git rev-list --no-walk 54b6849bd9436c0f6fb2fe82937ed1820d40de34 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins2120086499822312694.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+10.gb5e3995.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.89,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.88,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+10.gb5e3995,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.2.3,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='4086247310'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-et_oz2kq
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-et_oz2kq
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 14a18dc0de5d5fd7737ecbadf9f6d7fa5d801b67
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (7.0.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+9.g14a18dc) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+9.g14a18dc-py3-none-any.whl size=118253 sha256=4365e482466d7a270b35bac56edad8de125d9d84a2a47517bf812dc72c1b282d
  Stored in directory: /tmp/pip-ephem-wheel-cache-vtzdha2o/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+9.g14a18dc
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+10.gb5e3995) (0.7.0+9.g14a18dc)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+10.gb5e3995) (1.1.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.10.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (7.0.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (0.12.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (5.4.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (6.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (8.1.3)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+10.gb5e3995) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.6.0+10.gb5e3995-py3-none-any.whl size=83382 sha256=b213d1e6f8cef0864d96fb5e29b94c1576e7061b8b05dfee9817ee71b5594914
  Stored in directory: /tmp/pip-ephem-wheel-cache-ok9yh29v/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.6.0+10.gb5e3995
    Uninstalling merlin-systems-0.6.0+10.gb5e3995:
      Successfully uninstalled merlin-systems-0.6.0+10.gb5e3995
Successfully installed merlin-systems-0.6.0+10.gb5e3995
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 83 items

tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  3%]

.                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  7%]

[  7%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ...F                            [ 13%]

tests/unit/systems/dag/test_graph.py ..                                  [ 15%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 18%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 22%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 25%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 26%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 34%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 36%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 40%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 72%]

tests/unit/systems/ops/implicit/test_op.py ......                        [ 79%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 80%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 87%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 92%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

__________________________ test_triton_executor_model __________________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_triton_executor_model(tmpdir):
    request_schema = Schema(
        [
            ColumnSchema("candidate_ids", dtype=np.int32),
            ColumnSchema("movie_ids", dtype=np.int32),
        ]
    )

    candidate_ids = np.random.randint(1, 100000, 100).astype(np.int32)
    movie_ids_1 = np.zeros(100, dtype=np.int32)
    movie_ids_1[:20] = np.unique(candidate_ids)[:20]

    combined_features = {
        "candidate_ids": candidate_ids,
        "movie_ids": movie_ids_1,
    }

    request_data = DictArray(combined_features)

    filtering = ["candidate_ids"] >> FilterCandidates(filter_out=["movie_ids"])

    ensemble = Ensemble(filtering, request_schema)
    ensemble.export(tmpdir, backend="executor")





  response = run_ensemble_on_tritonserver(





        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        "executor_model",
    )



tests/unit/systems/dag/test_executors.py:157:




merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1007 19:16:27.405445 1863 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f5466000000' with size 268435456

I1007 19:16:27.406206 1863 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1007 19:16:27.409532 1863 model_lifecycle.cc:459] loading: executor_model:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1007 19:16:31.805673 1863 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

1007 19:16:34.707030 1942 pb_stub.cc:309] Failed to initialize Python stub: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0/1/ensemble/metadata.json'


At:

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(259): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(254): init

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(159): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/spec.py(1037): open

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(119): load

/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0/executor_model/1/model.py(66): initialize


/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

E1007 19:16:35.264033 1863 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0/1/ensemble/metadata.json'


At:

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(259): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(254): init

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(159): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/spec.py(1037): open

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(119): load

/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0/executor_model/1/model.py(66): initialize


I1007 19:16:35.264276 1863 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1007 19:16:35.264400 1863 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1007 19:16:35.264520 1863 server.cc:631]

+----------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model          | Version | Status                                                                                                                                                                 |

+----------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| executor_model | 1       | UNAVAILABLE: Internal: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0/1/ensemble/metadata.json' |

|                |         |                                                                                                                                                                        |

|                |         | At:                                                                                                                                                                    |

|                |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(259): _open                                                                     |

|                |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(254): init                                                                  |

|                |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(159): _open                                                                     |

|                |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/spec.py(1037): open                                                                                      |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(119): load                               |

|                |         |   /tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0/executor_model/1/model.py(66): initialize                                                               |

+----------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1007 19:16:35.328718 1863 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1007 19:16:35.329569 1863 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-25/test_triton_executor_model0                                                                                                                                 |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1007 19:16:35.329608 1863 server.cc:262] Waiting for in-flight requests to complete.

I1007 19:16:35.329615 1863 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1007 19:16:35.329622 1863 server.cc:293] All models are stopped, unloading models

I1007 19:16:35.329628 1863 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1007 19:16:36.354910 1863 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

_____________________ test_faiss_in_triton_executor_model ______________________


tmpdir = local('/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
def test_faiss_in_triton_executor_model(tmpdir):
    # Simulate a user vector with a TF model
    model = tf.keras.models.Sequential(
        [
            tf.keras.Input(name="user_id", dtype=tf.int32, shape=(1,)),
            tf.keras.layers.Dense(128, activation="relu", name="output"),
        ]
    )

    model.compile(
        optimizer="adam",
        loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[tf.metrics.SparseCategoricalAccuracy()],
    )

    faiss_path = tmpdir / "faiss.index"
    item_ids = np.arange(0, 100).reshape(-1, 1)
    item_embeddings = np.ascontiguousarray(np.random.rand(100, 128))
    setup_faiss(np.concatenate((item_ids, item_embeddings), axis=1), faiss_path)

    request_schema = Schema(
        [
            ColumnSchema("user_id", dtype=np.int32),
        ]
    )

    request_features = {
        "user_id": np.array([1]),
    }
    request_data = DictArray(request_features)

    filtering = ["user_id"] >> PredictTensorflow(model) >> QueryFaiss(faiss_path)

    ensemble = Ensemble(filtering, request_schema)
    ensemble.export(tmpdir, backend="executor")





  response = run_ensemble_on_tritonserver(





        tmpdir,
        ensemble.input_schema,
        make_df(request_data.arrays),
        ensemble.output_schema.column_names,
        "executor_model",
    )



tests/unit/systems/ops/faiss/test_executor.py:79:




merlin/systems/triton/utils.py:138: in run_ensemble_on_tritonserver

with run_triton_server(tmpdir) as client:

/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

WARNING clustering 100 points to 32 centroids: please provide at least 1248 training points

I1007 19:17:06.301905 2358 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f61e6000000' with size 268435456

I1007 19:17:06.302625 2358 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1007 19:17:06.306459 2358 model_lifecycle.cc:459] loading: 0_predicttensorflow:1

I1007 19:17:06.306500 2358 model_lifecycle.cc:459] loading: executor_model:1

I1007 19:17:06.589364 2358 tensorflow.cc:2536] TRITONBACKEND_Initialize: tensorflow

I1007 19:17:06.589398 2358 tensorflow.cc:2546] Triton TRITONBACKEND API version: 1.10

I1007 19:17:06.589405 2358 tensorflow.cc:2552] 'tensorflow' TRITONBACKEND API version: 1.10

I1007 19:17:06.589411 2358 tensorflow.cc:2576] backend configuration:

{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}}

I1007 19:17:06.589448 2358 tensorflow.cc:2642] TRITONBACKEND_ModelInitialize: 0_predicttensorflow (version 1)

2022-10-07 19:17:06.589989: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/0_predicttensorflow/1/model.savedmodel

2022-10-07 19:17:06.590779: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-07 19:17:06.590805: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/0_predicttensorflow/1/model.savedmodel

2022-10-07 19:17:06.590895: I tensorflow/core/platform/cpu_feature_guard.cc:194] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-07 19:17:07.012989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6251 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-07 19:17:07.045089: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled

2022-10-07 19:17:07.045690: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-07 19:17:07.066265: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/0_predicttensorflow/1/model.savedmodel

2022-10-07 19:17:07.073334: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 483356 microseconds.

I1007 19:17:07.079189 2358 tensorflow.cc:2691] TRITONBACKEND_ModelInstanceInitialize: 0_predicttensorflow (GPU device 0)

2022-10-07 19:17:07.079669: I tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/0_predicttensorflow/1/model.savedmodel

2022-10-07 19:17:07.080351: I tensorflow/cc/saved_model/reader.cc:81] Reading meta graph with tags { serve }

2022-10-07 19:17:07.080372: I tensorflow/cc/saved_model/reader.cc:122] Reading SavedModel debug info (if present) from: /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/0_predicttensorflow/1/model.savedmodel

2022-10-07 19:17:07.082255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6251 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-07 19:17:07.083762: I tensorflow/cc/saved_model/loader.cc:230] Restoring SavedModel bundle.

2022-10-07 19:17:07.105239: I tensorflow/cc/saved_model/loader.cc:214] Running initialization op on SavedModel bundle at path: /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/0_predicttensorflow/1/model.savedmodel

2022-10-07 19:17:07.112411: I tensorflow/cc/saved_model/loader.cc:321] SavedModel load for tags { serve }; Status: success: OK. Took 32745 microseconds.

I1007 19:17:07.112638 2358 model_lifecycle.cc:693] successfully loaded '0_predicttensorflow' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1007 19:17:11.485188 2358 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: executor_model (GPU device 0)

1007 19:17:14.439529 2475 pb_stub.cc:309] Failed to initialize Python stub: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/1/ensemble/metadata.json'


At:

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(259): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(254): init

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(159): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/spec.py(1037): open

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(119): load

/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/executor_model/1/model.py(66): initialize


/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

E1007 19:17:14.962662 2358 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/1/ensemble/metadata.json'


At:

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(259): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(254): init

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(159): _open

/var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/spec.py(1037): open

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(119): load

/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/executor_model/1/model.py(66): initialize


I1007 19:17:14.962860 2358 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1007 19:17:14.962949 2358 server.cc:588]

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend    | Path                                                            | Config                                                                                                                                                                      |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| tensorflow | /opt/tritonserver/backends/tensorflow2/libtriton_tensorflow2.so | {"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","version":"2","default-max-batch-size":"4"}} |

| python     | /opt/tritonserver/backends/python/libtriton_python.so           | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}}               |

+------------+-----------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1007 19:17:14.963055 2358 server.cc:631]

+---------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model               | Version | Status                                                                                                                                                                     |

+---------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predicttensorflow | 1       | READY                                                                                                                                                                      |

| executor_model      | 1       | UNAVAILABLE: Internal: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/1/ensemble/metadata.json' |

|                     |         |                                                                                                                                                                            |

|                     |         | At:                                                                                                                                                                        |

|                     |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(259): _open                                                                         |

|                     |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(254): init                                                                      |

|                     |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/implementations/local.py(159): _open                                                                         |

|                     |         |   /var/jenkins_home/.local/lib/python3.8/site-packages/fsspec/spec.py(1037): open                                                                                          |

|                     |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(119): load                                   |

|                     |         |   /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0/executor_model/1/model.py(66): initialize                                                               |

+---------------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1007 19:17:15.024700 2358 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1007 19:17:15.025557 2358 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-25/test_faiss_in_triton_executor_0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1007 19:17:15.025593 2358 server.cc:262] Waiting for in-flight requests to complete.

I1007 19:17:15.025601 2358 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1007 19:17:15.025643 2358 server.cc:293] All models are stopped, unloading models

I1007 19:17:15.025652 2358 server.cc:300] Timeout 30: Found 1 live models and 0 in-flight non-inference requests

I1007 19:17:15.025772 2358 tensorflow.cc:2729] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1007 19:17:15.026134 2358 tensorflow.cc:2668] TRITONBACKEND_ModelFinalize: delete model state

I1007 19:17:15.030170 2358 model_lifecycle.cc:578] successfully unloaded '0_predicttensorflow' version 1

I1007 19:17:16.025740 2358 server.cc:300] Timeout 29: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1007 19:17:16.043003 2358 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  221     33    85%

merlin/systems/dag/ops/implicit.py              58      8    86%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2272    879    61%


============ 2 failed, 81 passed, 56 warnings in 523.35s (0:08:43) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins2704453289343389292.sh

oliverholworthy
oliverholworthy reviewed Oct 7, 2022
View reviewed changes
merlin/systems/dag/ops/faiss.py

# set index path to new path after export
full_path = pathlib.Path(path) / f"{node_id}_{QueryFaiss.__name__.lower()}" / str(version)
if backend == "ensemble":
Copy link
Contributor

@oliverholworthy oliverholworthy Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this conditional export only apply to the Faiss op and not all the others?

Copy link
Contributor Author

@karlhigley karlhigley Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QueryFAISS is the only Python op currently saving additional files to disk (aside from the Triton config and model.py.) In an ensemble export, we write the FAISS index file into the n_queryfaiss directory in the model repository, but that directory doesn't exist with an executor export, so instead we write it to the executor_model directory. If other ops were also writing additional files, this conditional would apply there too.

(But should disappear when we introduce backends/runtimes/adapters, since then we can have separate export methods for each export format.)

Copy link
Contributor

@oliverholworthy oliverholworthy Oct 7, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Implicit op also writes out files to it's directory. Is that a different kind of file in this context?

Copy link
Contributor Author

@karlhigley karlhigley Oct 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good point! That should probably write its files to the executor_model directory too then.

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 19, 2022

Click to view CI Results
GitHub pull request #204 of commit dfbe43a1afb71bc0b1307d2db739a5cd68db4df6, no merge conflicts.
Running as SYSTEM
Setting status of dfbe43a1afb71bc0b1307d2db739a5cd68db4df6 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/436/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse dfbe43a1afb71bc0b1307d2db739a5cd68db4df6^{commit} # timeout=10
Checking out Revision dfbe43a1afb71bc0b1307d2db739a5cd68db4df6 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f dfbe43a1afb71bc0b1307d2db739a5cd68db4df6 # timeout=10
Commit message: "Fix the artifact path in the executor model"
 > git rev-list --no-walk b84e881e183bd71171fa62f69327132e16cef6b2 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins3172954756476421755.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+13.gdfbe43a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.94,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.93,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+13.gdfbe43a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3497762360'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-g22ry9_l
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-g22ry9_l
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 14a18dc0de5d5fd7737ecbadf9f6d7fa5d801b67
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (4.64.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (0.55.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (7.0.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+9.g14a18dc) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+9.g14a18dc) (3.19.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.2.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.0.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.4.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+9.g14a18dc) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+9.g14a18dc) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+9.g14a18dc) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+9.g14a18dc) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+9.g14a18dc) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+9.g14a18dc-py3-none-any.whl size=118253 sha256=2a27ba454432427d6f51fe83f48281914b4d012bbdf6e9241514892b341f8c1b
  Stored in directory: /tmp/pip-ephem-wheel-cache-1ws81vfm/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+9.g14a18dc

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+13.gdfbe43a) (0.7.0+9.g14a18dc)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+13.gdfbe43a) (1.1.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (4.64.1)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.2.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.3.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (0.55.1)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (7.0.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2022.5.0)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.10.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (3.19.5)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (0.4.3)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (5.4.1)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (0.12.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2.2.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.7.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2.0.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.0.4)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2.4.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (8.1.3)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (3.1.2)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (65.3.0)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (0.38.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.20.3)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2022.2.1)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.2.0)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.52.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+13.gdfbe43a) (6.0.1)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+13.gdfbe43a-py3-none-any.whl size=83546 sha256=20ed007b27d80d3321f05f066723ae79b864bea3886a577210c2505c3ca9bf19

Stored in directory: /tmp/pip-ephem-wheel-cache-8xn4gasg/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+13.gdfbe43a

Uninstalling merlin-systems-0.6.0+13.gdfbe43a:

Successfully uninstalled merlin-systems-0.6.0+13.gdfbe43a

Successfully installed merlin-systems-0.6.0+13.gdfbe43a


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 87 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/test_version.py .                                             [  2%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py F [  3%]

F                                                                        [  4%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py F [  5%]

[  5%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  6%]

[  6%]

tests/unit/systems/dag/test_ensemble.py .                                [  8%]

tests/unit/systems/dag/test_executors.py ....                            [ 12%]

tests/unit/systems/dag/test_graph.py ..                                  [ 14%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 17%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 21%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 24%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 25%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 33%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 34%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 39%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 70%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_____________________ test_example_serving_implicit[True] ______________________


self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'markdown', 'id': '76dad9c3', 'metadata': {}, 'source': 'With inference operations defined, all that remains now is outputting the ensemble to disk so that it can be loaded up when Triton starts.'}

kwargs = {}, cell_indexes = [0, 1, 2, 3, 4, 5, ...]

executed_cells = [{'cell_type': 'code', 'execution_count': 2, 'id': '5cdba80f', 'metadata': {'execution': {'iopub.status.busy': '2022-1...: 'markdown', 'id': 'ff168b4a', 'metadata': {}, 'source': "Having preprocessed our data, let's train our model."}, ...]

idx = 13


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>, {'cell_type': 'code', 'execution_count': 7, 'id': ...port_path = 'ensemble'\n\nensemble = Ensemble(inf_ops, inf_workflow.input_schema)\nensemble.export(export_path);"}, 13)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7feb404d49c0>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-36' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib/...mn 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'code', 'execution_count': 7, 'id': 'e23a7fc3', 'metadata': {'execution': {'iopub.status.busy': '2022-10...: "export_path = 'ensemble'\n\nensemble = Ensemble(inf_ops, inf_workflow.input_schema)\nensemble.export(export_path);"}

cell_index = 13, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'code', 'execution_count': 7, 'id': 'e23a7fc3', 'metadata': {'execution': {'iopub.status.busy': '2022-10...: "export_path = 'ensemble'\n\nensemble = Ensemble(inf_ops, inf_workflow.input_schema)\nensemble.export(export_path);"}

cell_index = 13

exec_reply = {'buffers': [], 'content': {'ename': 'ValueError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'd01357d8-b138-49aa...e, 'engine': 'd01357d8-b138-49aa-bee0-2a47cd036cc2', 'started': '2022-10-19T15:26:54.364249Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           export_path = 'ensemble'

E

E           ensemble = Ensemble(inf_ops, inf_workflow.input_schema)

E           ensemble.export(export_path);

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mValueError�[0m                                Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 3�[0m

E           �[1;32m      1�[0m export_path �[38;5;241m=�[39m �[38;5;124m'�[39m�[38;5;124mensemble�[39m�[38;5;124m'�[39m

E           �[0;32m----> 3�[0m ensemble �[38;5;241m=�[39m �[43mEnsemble�[49m�[43m(�[49m�[43minf_ops�[49m�[43m,�[49m�[43m �[49m�[43minf_workflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m)�[49m

E           �[1;32m      4�[0m ensemble�[38;5;241m.�[39mexport(export_path)

E

E           File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/ensemble.py:61�[0m, in �[0;36mEnsemble.__init__�[0;34m(self, ops, schema, name, label_columns)�[0m

E           �[1;32m     47�[0m �[38;5;124;03m"""_summary_�[39;00m

E           �[1;32m     48�[0m

E           �[1;32m     49�[0m �[38;5;124;03mParameters�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     58�[0m �[38;5;124;03m    List of strings representing label columns, by default None�[39;00m

E           �[1;32m     59�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m     60�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mgraph �[38;5;241m=�[39m Graph(ops)

E           �[0;32m---> 61�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43mconstruct_schema�[49m�[43m(�[49m�[43mschema�[49m�[43m)�[49m

E           �[1;32m     62�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mname �[38;5;241m=�[39m name

E           �[1;32m     63�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mlabel_columns �[38;5;241m=�[39m label_columns �[38;5;129;01mor�[39;00m []

E

E           File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:90�[0m, in �[0;36mGraph.construct_schema�[0;34m(self, root_schema, preserve_dtypes)�[0m

E           �[1;32m     87�[0m nodes �[38;5;241m=�[39m �[38;5;28mlist�[39m(postorder_iter_nodes(�[38;5;28mself�[39m�[38;5;241m.�[39moutput_node))

E           �[1;32m     89�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_compute_node_schemas(root_schema, nodes, preserve_dtypes)

E           �[0;32m---> 90�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_validate_node_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mnodes�[49m�[43m,�[49m�[43m �[49m�[43mpreserve_dtypes�[49m�[43m)�[49m

E           �[1;32m     92�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:100�[0m, in �[0;36mGraph._validate_node_schemas�[0;34m(self, root_schema, nodes, strict_dtypes)�[0m

E           �[1;32m     98�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_validate_node_schemas�[39m(�[38;5;28mself�[39m, root_schema, nodes, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E           �[1;32m     99�[0m     �[38;5;28;01mfor�[39;00m node �[38;5;129;01min�[39;00m nodes:

E           �[0;32m--> 100�[0m         �[43mnode�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[38;5;241;43m=�[39;49m�[43mstrict_dtypes�[49m�[43m)�[49m

E

E           File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/node.py:109�[0m, in �[0;36mInferenceNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E           �[1;32m     90�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mvalidate_schemas�[39m(�[38;5;28mself�[39m, root_schema, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E           �[1;32m     91�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     92�[0m �[38;5;124;03m    Checks that the output schema is valid given the previous�[39;00m

E           �[1;32m     93�[0m �[38;5;124;03m    nodes in the graph and following nodes in the graph, as�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    107�[0m �[38;5;124;03m        If an output column is produced but not used by child nodes�[39;00m

E           �[1;32m    108�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m--> 109�[0m     �[38;5;28;43msuper�[39;49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[43m)�[49m

E           �[1;32m    111�[0m     �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mchildren:

E           �[1;32m    112�[0m         childrens_schema �[38;5;241m=�[39m Schema()

E

E           File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py:226�[0m, in �[0;36mNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E           �[1;32m    224�[0m �[38;5;28;01mif�[39;00m strict_dtypes �[38;5;129;01mor�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mdynamic_dtypes:

E           �[1;32m    225�[0m     �[38;5;28;01mif�[39;00m source_col_schema�[38;5;241m.�[39mdtype �[38;5;241m!=�[39m col_schema�[38;5;241m.�[39mdtype:

E           �[0;32m--> 226�[0m         �[38;5;28;01mraise�[39;00m �[38;5;167;01mValueError�[39;00m(

E           �[1;32m    227�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mMismatched dtypes for column �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_name�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m provided to �[39m�[38;5;124m"�[39m

E           �[1;32m    228�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00m�[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39m�[38;5;18m__class__�[39m�[38;5;241m.�[39m�[38;5;18m__name__�[39m�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m: �[39m�[38;5;124m"�[39m

E           �[1;32m    229�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mancestor nodes provided dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00msource_col_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m, �[39m�[38;5;124m"�[39m

E           �[1;32m    230�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mexpected dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m.�[39m�[38;5;124m"�[39m

E           �[1;32m    231�[0m         )

E           �[1;32m    233�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mvalidate_schemas(

E           �[1;32m    234�[0m     parents_schema, deps_schema, �[38;5;28mself�[39m�[38;5;241m.�[39minput_schema, �[38;5;28mself�[39m�[38;5;241m.�[39moutput_schema, strict_dtypes

E           �[1;32m    235�[0m )

E

E           �[0;31mValueError�[0m: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.

E           ValueError: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


tb = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

gpu = True


@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
@testbook(REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_implicit(tb, gpu):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """,
        pop=True,
    )
    NUM_OF_CELLS = len(tb.cells)
    if not gpu:
        tb.cells[
            6
        ].source = (
            "model = BayesianPersonalizedRanking(use_gpu=False)\nmodel.fit(train_transformed)"
        )





  tb.execute_cell(list(range(0, 16)))





tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:52:




self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'markdown', 'id': '76dad9c3', 'metadata': {}, 'source': 'With inference operations defined, all that remains now is outputting the ensemble to disk so that it can be loaded up when Triton starts.'}

kwargs = {}, cell_indexes = [0, 1, 2, 3, 4, 5, ...]

executed_cells = [{'cell_type': 'code', 'execution_count': 2, 'id': '5cdba80f', 'metadata': {'execution': {'iopub.status.busy': '2022-1...: 'markdown', 'id': 'ff168b4a', 'metadata': {}, 'source': "Having preprocessed our data, let's train our model."}, ...]

idx = 13


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E               export_path = 'ensemble'

E

E               ensemble = Ensemble(inf_ops, inf_workflow.input_schema)

E               ensemble.export(export_path);

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mValueError�[0m                                Traceback (most recent call last)

E               Cell �[0;32mIn [7], line 3�[0m

E               �[1;32m      1�[0m export_path �[38;5;241m=�[39m �[38;5;124m'�[39m�[38;5;124mensemble�[39m�[38;5;124m'�[39m

E               �[0;32m----> 3�[0m ensemble �[38;5;241m=�[39m �[43mEnsemble�[49m�[43m(�[49m�[43minf_ops�[49m�[43m,�[49m�[43m �[49m�[43minf_workflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m)�[49m

E               �[1;32m      4�[0m ensemble�[38;5;241m.�[39mexport(export_path)

E

E               File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/ensemble.py:61�[0m, in �[0;36mEnsemble.__init__�[0;34m(self, ops, schema, name, label_columns)�[0m

E               �[1;32m     47�[0m �[38;5;124;03m"""_summary_�[39;00m

E               �[1;32m     48�[0m

E               �[1;32m     49�[0m �[38;5;124;03mParameters�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m     58�[0m �[38;5;124;03m    List of strings representing label columns, by default None�[39;00m

E               �[1;32m     59�[0m �[38;5;124;03m"""�[39;00m

E               �[1;32m     60�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mgraph �[38;5;241m=�[39m Graph(ops)

E               �[0;32m---> 61�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43mconstruct_schema�[49m�[43m(�[49m�[43mschema�[49m�[43m)�[49m

E               �[1;32m     62�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mname �[38;5;241m=�[39m name

E               �[1;32m     63�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mlabel_columns �[38;5;241m=�[39m label_columns �[38;5;129;01mor�[39;00m []

E

E               File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:90�[0m, in �[0;36mGraph.construct_schema�[0;34m(self, root_schema, preserve_dtypes)�[0m

E               �[1;32m     87�[0m nodes �[38;5;241m=�[39m �[38;5;28mlist�[39m(postorder_iter_nodes(�[38;5;28mself�[39m�[38;5;241m.�[39moutput_node))

E               �[1;32m     89�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_compute_node_schemas(root_schema, nodes, preserve_dtypes)

E               �[0;32m---> 90�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_validate_node_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mnodes�[49m�[43m,�[49m�[43m �[49m�[43mpreserve_dtypes�[49m�[43m)�[49m

E               �[1;32m     92�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E               File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:100�[0m, in �[0;36mGraph._validate_node_schemas�[0;34m(self, root_schema, nodes, strict_dtypes)�[0m

E               �[1;32m     98�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_validate_node_schemas�[39m(�[38;5;28mself�[39m, root_schema, nodes, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E               �[1;32m     99�[0m     �[38;5;28;01mfor�[39;00m node �[38;5;129;01min�[39;00m nodes:

E               �[0;32m--> 100�[0m         �[43mnode�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[38;5;241;43m=�[39;49m�[43mstrict_dtypes�[49m�[43m)�[49m

E

E               File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/node.py:109�[0m, in �[0;36mInferenceNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E               �[1;32m     90�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mvalidate_schemas�[39m(�[38;5;28mself�[39m, root_schema, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E               �[1;32m     91�[0m     �[38;5;124;03m"""�[39;00m

E               �[1;32m     92�[0m �[38;5;124;03m    Checks that the output schema is valid given the previous�[39;00m

E               �[1;32m     93�[0m �[38;5;124;03m    nodes in the graph and following nodes in the graph, as�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m    107�[0m �[38;5;124;03m        If an output column is produced but not used by child nodes�[39;00m

E               �[1;32m    108�[0m �[38;5;124;03m    """�[39;00m

E               �[0;32m--> 109�[0m     �[38;5;28;43msuper�[39;49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[43m)�[49m

E               �[1;32m    111�[0m     �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mchildren:

E               �[1;32m    112�[0m         childrens_schema �[38;5;241m=�[39m Schema()

E

E               File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py:226�[0m, in �[0;36mNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E               �[1;32m    224�[0m �[38;5;28;01mif�[39;00m strict_dtypes �[38;5;129;01mor�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mdynamic_dtypes:

E               �[1;32m    225�[0m     �[38;5;28;01mif�[39;00m source_col_schema�[38;5;241m.�[39mdtype �[38;5;241m!=�[39m col_schema�[38;5;241m.�[39mdtype:

E               �[0;32m--> 226�[0m         �[38;5;28;01mraise�[39;00m �[38;5;167;01mValueError�[39;00m(

E               �[1;32m    227�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mMismatched dtypes for column �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_name�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m provided to �[39m�[38;5;124m"�[39m

E               �[1;32m    228�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00m�[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39m�[38;5;18m__class__�[39m�[38;5;241m.�[39m�[38;5;18m__name__�[39m�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m: �[39m�[38;5;124m"�[39m

E               �[1;32m    229�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mancestor nodes provided dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00msource_col_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m, �[39m�[38;5;124m"�[39m

E               �[1;32m    230�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mexpected dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m.�[39m�[38;5;124m"�[39m

E               �[1;32m    231�[0m         )

E               �[1;32m    233�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mvalidate_schemas(

E               �[1;32m    234�[0m     parents_schema, deps_schema, �[38;5;28mself�[39m�[38;5;241m.�[39minput_schema, �[38;5;28mself�[39m�[38;5;241m.�[39moutput_schema, strict_dtypes

E               �[1;32m    235�[0m )

E

E               �[0;31mValueError�[0m: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.

E               ValueError: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

_____________________ test_example_serving_implicit[False] _____________________


self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'markdown', 'id': '76dad9c3', 'metadata': {}, 'source': 'With inference operations defined, all that remains now is outputting the ensemble to disk so that it can be loaded up when Triton starts.'}

kwargs = {}, cell_indexes = [0, 1, 2, 3, 4, 5, ...]

executed_cells = [{'cell_type': 'code', 'execution_count': 2, 'id': '5cdba80f', 'metadata': {'execution': {'iopub.status.busy': '2022-1...: 'markdown', 'id': 'ff168b4a', 'metadata': {}, 'source': "Having preprocessed our data, let's train our model."}, ...]

idx = 13


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:





          cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)





../../../.local/lib/python3.8/site-packages/testbook/client.py:133:




args = (<testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>, {'cell_type': 'code', 'execution_count': 7, 'id': ...port_path = 'ensemble'\n\nensemble = Ensemble(inf_ops, inf_workflow.input_schema)\nensemble.export(export_path);"}, 13)

kwargs = {}


def wrapped(*args, **kwargs):





  return just_run(coro(*args, **kwargs))





../../../.local/lib/python3.8/site-packages/nbclient/util.py:84:




coro = <coroutine object NotebookClient.async_execute_cell at 0x7feb4043a9c0>


def just_run(coro: Awaitable) -> Any:
    """Make the coroutine run, even if there is an event loop running (using nest_asyncio)"""
    # original from vaex/asyncio.py
    loop = asyncio._get_running_loop()
    if loop is None:
        had_running_loop = False
        try:
            loop = asyncio.get_event_loop()
        except RuntimeError:
            # we can still get 'There is no current event loop in ...'
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
    else:
        had_running_loop = True
    if had_running_loop:
        # if there is a running loop, we patch using nest_asyncio
        # to have reentrant event loops
        check_ipython()
        import nest_asyncio

        nest_asyncio.apply()
        check_patch_tornado()





  return loop.run_until_complete(coro)





../../../.local/lib/python3.8/site-packages/nbclient/util.py:62:




self = <_UnixSelectorEventLoop running=False closed=False debug=False>

future = <Task finished name='Task-78' coro=<NotebookClient.async_execute_cell() done, defined at /var/jenkins_home/.local/lib/...mn 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.\n')>


def run_until_complete(self, future):
    """Run until the Future is done.

    If the argument is a coroutine, it is wrapped in a Task.

    WARNING: It would be disastrous to call run_until_complete()
    with the same coroutine twice -- it would wrap it in two
    different Tasks and that can't be good.

    Return the Future's result, or raise its exception.
    """
    self._check_closed()
    self._check_running()

    new_task = not futures.isfuture(future)
    future = tasks.ensure_future(future, loop=self)
    if new_task:
        # An exception is raised if the future didn't complete, so there
        # is no need to log the "destroy pending task" message
        future._log_destroy_pending = False

    future.add_done_callback(_run_until_complete_cb)
    try:
        self.run_forever()
    except:
        if new_task and future.done() and not future.cancelled():
            # The coroutine raised a BaseException. Consume the exception
            # to not log a warning, the caller doesn't have access to the
            # local task.
            future.exception()
        raise
    finally:
        future.remove_done_callback(_run_until_complete_cb)
    if not future.done():
        raise RuntimeError('Event loop stopped before Future completed.')





  return future.result()





/usr/lib/python3.8/asyncio/base_events.py:616:




self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'code', 'execution_count': 7, 'id': 'e23a7fc3', 'metadata': {'execution': {'iopub.status.busy': '2022-10...: "export_path = 'ensemble'\n\nensemble = Ensemble(inf_ops, inf_workflow.input_schema)\nensemble.export(export_path);"}

cell_index = 13, execution_count = None, store_history = True


async def async_execute_cell(
    self,
    cell: NotebookNode,
    cell_index: int,
    execution_count: t.Optional[int] = None,
    store_history: bool = True,
) -> NotebookNode:
    """
    Executes a single code cell.

    To execute all cells see :meth:`execute`.

    Parameters
    ----------
    cell : nbformat.NotebookNode
        The cell which is currently being processed.
    cell_index : int
        The position of the cell within the notebook object.
    execution_count : int
        The execution count to be assigned to the cell (default: Use kernel response)
    store_history : bool
        Determines if history should be stored in the kernel (default: False).
        Specific to ipython kernels, which can store command histories.

    Returns
    -------
    output : dict
        The execution output payload (or None for no output).

    Raises
    ------
    CellExecutionError
        If execution failed and should raise an exception, this will be raised
        with defaults about the failure.

    Returns
    -------
    cell : NotebookNode
        The cell which was just processed.
    """
    assert self.kc is not None

    await run_hook(self.on_cell_start, cell=cell, cell_index=cell_index)

    if cell.cell_type != 'code' or not cell.source.strip():
        self.log.debug("Skipping non-executing cell %s", cell_index)
        return cell

    if self.skip_cells_with_tag in cell.metadata.get("tags", []):
        self.log.debug("Skipping tagged cell %s", cell_index)
        return cell

    if self.record_timing:  # clear execution metadata prior to execution
        cell['metadata']['execution'] = {}

    self.log.debug("Executing cell:\n%s", cell.source)

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors or "raises-exception" in cell.metadata.get("tags", [])
    )

    await run_hook(self.on_cell_execute, cell=cell, cell_index=cell_index)
    parent_msg_id = await ensure_async(
        self.kc.execute(
            cell.source, store_history=store_history, stop_on_error=not cell_allows_errors
        )
    )
    await run_hook(self.on_cell_complete, cell=cell, cell_index=cell_index)
    # We launched a code cell to execute
    self.code_cells_executed += 1
    exec_timeout = self._get_timeout(cell)

    cell.outputs = []
    self.clear_before_next_output = False

    task_poll_kernel_alive = asyncio.ensure_future(self._async_poll_kernel_alive())
    task_poll_output_msg = asyncio.ensure_future(
        self._async_poll_output_msg(parent_msg_id, cell, cell_index)
    )
    self.task_poll_for_reply = asyncio.ensure_future(
        self._async_poll_for_reply(
            parent_msg_id, cell, exec_timeout, task_poll_output_msg, task_poll_kernel_alive
        )
    )
    try:
        exec_reply = await self.task_poll_for_reply
    except asyncio.CancelledError:
        # can only be cancelled by task_poll_kernel_alive when the kernel is dead
        task_poll_output_msg.cancel()
        raise DeadKernelError("Kernel died")
    except Exception as e:
        # Best effort to cancel request if it hasn't been resolved
        try:
            # Check if the task_poll_output is doing the raising for us
            if not isinstance(e, CellControlSignal):
                task_poll_output_msg.cancel()
        finally:
            raise

    if execution_count:
        cell['execution_count'] = execution_count





  await self._check_raise_for_error(cell, cell_index, exec_reply)





../../../.local/lib/python3.8/site-packages/nbclient/client.py:965:




self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'code', 'execution_count': 7, 'id': 'e23a7fc3', 'metadata': {'execution': {'iopub.status.busy': '2022-10...: "export_path = 'ensemble'\n\nensemble = Ensemble(inf_ops, inf_workflow.input_schema)\nensemble.export(export_path);"}

cell_index = 13

exec_reply = {'buffers': [], 'content': {'ename': 'ValueError', 'engine_info': {'engine_id': -1, 'engine_uuid': 'e0cd0c77-c75f-421c...e, 'engine': 'e0cd0c77-c75f-421c-babb-0a44520b0148', 'started': '2022-10-19T15:27:03.567866Z', 'status': 'error'}, ...}


async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(self.on_cell_error, cell=cell, cell_index=cell_index)
    if not cell_allows_errors:





      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)





E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           export_path = 'ensemble'

E

E           ensemble = Ensemble(inf_ops, inf_workflow.input_schema)

E           ensemble.export(export_path);

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mValueError�[0m                                Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 3�[0m

E           �[1;32m      1�[0m export_path �[38;5;241m=�[39m �[38;5;124m'�[39m�[38;5;124mensemble�[39m�[38;5;124m'�[39m

E           �[0;32m----> 3�[0m ensemble �[38;5;241m=�[39m �[43mEnsemble�[49m�[43m(�[49m�[43minf_ops�[49m�[43m,�[49m�[43m �[49m�[43minf_workflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m)�[49m

E           �[1;32m      4�[0m ensemble�[38;5;241m.�[39mexport(export_path)

E

E           File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/ensemble.py:61�[0m, in �[0;36mEnsemble.__init__�[0;34m(self, ops, schema, name, label_columns)�[0m

E           �[1;32m     47�[0m �[38;5;124;03m"""_summary_�[39;00m

E           �[1;32m     48�[0m

E           �[1;32m     49�[0m �[38;5;124;03mParameters�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m     58�[0m �[38;5;124;03m    List of strings representing label columns, by default None�[39;00m

E           �[1;32m     59�[0m �[38;5;124;03m"""�[39;00m

E           �[1;32m     60�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mgraph �[38;5;241m=�[39m Graph(ops)

E           �[0;32m---> 61�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43mconstruct_schema�[49m�[43m(�[49m�[43mschema�[49m�[43m)�[49m

E           �[1;32m     62�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mname �[38;5;241m=�[39m name

E           �[1;32m     63�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mlabel_columns �[38;5;241m=�[39m label_columns �[38;5;129;01mor�[39;00m []

E

E           File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:90�[0m, in �[0;36mGraph.construct_schema�[0;34m(self, root_schema, preserve_dtypes)�[0m

E           �[1;32m     87�[0m nodes �[38;5;241m=�[39m �[38;5;28mlist�[39m(postorder_iter_nodes(�[38;5;28mself�[39m�[38;5;241m.�[39moutput_node))

E           �[1;32m     89�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_compute_node_schemas(root_schema, nodes, preserve_dtypes)

E           �[0;32m---> 90�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_validate_node_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mnodes�[49m�[43m,�[49m�[43m �[49m�[43mpreserve_dtypes�[49m�[43m)�[49m

E           �[1;32m     92�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E           File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:100�[0m, in �[0;36mGraph._validate_node_schemas�[0;34m(self, root_schema, nodes, strict_dtypes)�[0m

E           �[1;32m     98�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_validate_node_schemas�[39m(�[38;5;28mself�[39m, root_schema, nodes, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E           �[1;32m     99�[0m     �[38;5;28;01mfor�[39;00m node �[38;5;129;01min�[39;00m nodes:

E           �[0;32m--> 100�[0m         �[43mnode�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[38;5;241;43m=�[39;49m�[43mstrict_dtypes�[49m�[43m)�[49m

E

E           File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/node.py:109�[0m, in �[0;36mInferenceNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E           �[1;32m     90�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mvalidate_schemas�[39m(�[38;5;28mself�[39m, root_schema, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E           �[1;32m     91�[0m     �[38;5;124;03m"""�[39;00m

E           �[1;32m     92�[0m �[38;5;124;03m    Checks that the output schema is valid given the previous�[39;00m

E           �[1;32m     93�[0m �[38;5;124;03m    nodes in the graph and following nodes in the graph, as�[39;00m

E           �[0;32m   (...)�[0m

E           �[1;32m    107�[0m �[38;5;124;03m        If an output column is produced but not used by child nodes�[39;00m

E           �[1;32m    108�[0m �[38;5;124;03m    """�[39;00m

E           �[0;32m--> 109�[0m     �[38;5;28;43msuper�[39;49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[43m)�[49m

E           �[1;32m    111�[0m     �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mchildren:

E           �[1;32m    112�[0m         childrens_schema �[38;5;241m=�[39m Schema()

E

E           File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py:226�[0m, in �[0;36mNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E           �[1;32m    224�[0m �[38;5;28;01mif�[39;00m strict_dtypes �[38;5;129;01mor�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mdynamic_dtypes:

E           �[1;32m    225�[0m     �[38;5;28;01mif�[39;00m source_col_schema�[38;5;241m.�[39mdtype �[38;5;241m!=�[39m col_schema�[38;5;241m.�[39mdtype:

E           �[0;32m--> 226�[0m         �[38;5;28;01mraise�[39;00m �[38;5;167;01mValueError�[39;00m(

E           �[1;32m    227�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mMismatched dtypes for column �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_name�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m provided to �[39m�[38;5;124m"�[39m

E           �[1;32m    228�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00m�[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39m�[38;5;18m__class__�[39m�[38;5;241m.�[39m�[38;5;18m__name__�[39m�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m: �[39m�[38;5;124m"�[39m

E           �[1;32m    229�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mancestor nodes provided dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00msource_col_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m, �[39m�[38;5;124m"�[39m

E           �[1;32m    230�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mexpected dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m.�[39m�[38;5;124m"�[39m

E           �[1;32m    231�[0m         )

E           �[1;32m    233�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mvalidate_schemas(

E           �[1;32m    234�[0m     parents_schema, deps_schema, �[38;5;28mself�[39m�[38;5;241m.�[39minput_schema, �[38;5;28mself�[39m�[38;5;241m.�[39moutput_schema, strict_dtypes

E           �[1;32m    235�[0m )

E

E           �[0;31mValueError�[0m: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.

E           ValueError: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.


../../../.local/lib/python3.8/site-packages/nbclient/client.py:862: CellExecutionError


During handling of the above exception, another exception occurred:


tb = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

gpu = False


@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
@testbook(REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_implicit(tb, gpu):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """,
        pop=True,
    )
    NUM_OF_CELLS = len(tb.cells)
    if not gpu:
        tb.cells[
            6
        ].source = (
            "model = BayesianPersonalizedRanking(use_gpu=False)\nmodel.fit(train_transformed)"
        )





  tb.execute_cell(list(range(0, 16)))





tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:52:




self = <testbook.client.TestbookNotebookClient object at 0x7febc08c07c0>

cell = {'cell_type': 'markdown', 'id': '76dad9c3', 'metadata': {}, 'source': 'With inference operations defined, all that remains now is outputting the ensemble to disk so that it can be loaded up when Triton starts.'}

kwargs = {}, cell_indexes = [0, 1, 2, 3, 4, 5, ...]

executed_cells = [{'cell_type': 'code', 'execution_count': 2, 'id': '5cdba80f', 'metadata': {'execution': {'iopub.status.busy': '2022-1...: 'markdown', 'id': 'ff168b4a', 'metadata': {}, 'source': "Having preprocessed our data, let's train our model."}, ...]

idx = 13


def execute_cell(self, cell, **kwargs) -> Union[Dict, List[Dict]]:
    """
    Executes a cell or list of cells
    """
    if isinstance(cell, slice):
        start, stop = self._cell_index(cell.start), self._cell_index(cell.stop)
        if cell.step is not None:
            raise TestbookError('testbook does not support step argument')

        cell = range(start, stop + 1)
    elif isinstance(cell, str) or isinstance(cell, int):
        cell = [cell]

    cell_indexes = cell

    if all(isinstance(x, str) for x in cell):
        cell_indexes = [self._cell_index(tag) for tag in cell]

    executed_cells = []
    for idx in cell_indexes:
        try:
            cell = super().execute_cell(self.nb['cells'][idx], idx, **kwargs)
        except CellExecutionError as ce:





          raise TestbookRuntimeError(ce.evalue, ce, self._get_error_class(ce.ename))





E               testbook.exceptions.TestbookRuntimeError: An error occurred while executing the following cell:

E               ------------------

E               export_path = 'ensemble'

E

E               ensemble = Ensemble(inf_ops, inf_workflow.input_schema)

E               ensemble.export(export_path);

E               ------------------

E

E               �[0;31m---------------------------------------------------------------------------�[0m

E               �[0;31mValueError�[0m                                Traceback (most recent call last)

E               Cell �[0;32mIn [7], line 3�[0m

E               �[1;32m      1�[0m export_path �[38;5;241m=�[39m �[38;5;124m'�[39m�[38;5;124mensemble�[39m�[38;5;124m'�[39m

E               �[0;32m----> 3�[0m ensemble �[38;5;241m=�[39m �[43mEnsemble�[49m�[43m(�[49m�[43minf_ops�[49m�[43m,�[49m�[43m �[49m�[43minf_workflow�[49m�[38;5;241;43m.�[39;49m�[43minput_schema�[49m�[43m)�[49m

E               �[1;32m      4�[0m ensemble�[38;5;241m.�[39mexport(export_path)

E

E               File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/ensemble.py:61�[0m, in �[0;36mEnsemble.__init__�[0;34m(self, ops, schema, name, label_columns)�[0m

E               �[1;32m     47�[0m �[38;5;124;03m"""_summary_�[39;00m

E               �[1;32m     48�[0m

E               �[1;32m     49�[0m �[38;5;124;03mParameters�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m     58�[0m �[38;5;124;03m    List of strings representing label columns, by default None�[39;00m

E               �[1;32m     59�[0m �[38;5;124;03m"""�[39;00m

E               �[1;32m     60�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mgraph �[38;5;241m=�[39m Graph(ops)

E               �[0;32m---> 61�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43mgraph�[49m�[38;5;241;43m.�[39;49m�[43mconstruct_schema�[49m�[43m(�[49m�[43mschema�[49m�[43m)�[49m

E               �[1;32m     62�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mname �[38;5;241m=�[39m name

E               �[1;32m     63�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mlabel_columns �[38;5;241m=�[39m label_columns �[38;5;129;01mor�[39;00m []

E

E               File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:90�[0m, in �[0;36mGraph.construct_schema�[0;34m(self, root_schema, preserve_dtypes)�[0m

E               �[1;32m     87�[0m nodes �[38;5;241m=�[39m �[38;5;28mlist�[39m(postorder_iter_nodes(�[38;5;28mself�[39m�[38;5;241m.�[39moutput_node))

E               �[1;32m     89�[0m �[38;5;28mself�[39m�[38;5;241m.�[39m_compute_node_schemas(root_schema, nodes, preserve_dtypes)

E               �[0;32m---> 90�[0m �[38;5;28;43mself�[39;49m�[38;5;241;43m.�[39;49m�[43m_validate_node_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mnodes�[49m�[43m,�[49m�[43m �[49m�[43mpreserve_dtypes�[49m�[43m)�[49m

E               �[1;32m     92�[0m �[38;5;28;01mreturn�[39;00m �[38;5;28mself�[39m

E

E               File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/graph.py:100�[0m, in �[0;36mGraph._validate_node_schemas�[0;34m(self, root_schema, nodes, strict_dtypes)�[0m

E               �[1;32m     98�[0m �[38;5;28;01mdef�[39;00m �[38;5;21m_validate_node_schemas�[39m(�[38;5;28mself�[39m, root_schema, nodes, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E               �[1;32m     99�[0m     �[38;5;28;01mfor�[39;00m node �[38;5;129;01min�[39;00m nodes:

E               �[0;32m--> 100�[0m         �[43mnode�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[38;5;241;43m=�[39;49m�[43mstrict_dtypes�[49m�[43m)�[49m

E

E               File �[0;32m~/workspace/merlin_systems/systems/merlin/systems/dag/node.py:109�[0m, in �[0;36mInferenceNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E               �[1;32m     90�[0m �[38;5;28;01mdef�[39;00m �[38;5;21mvalidate_schemas�[39m(�[38;5;28mself�[39m, root_schema, strict_dtypes�[38;5;241m=�[39m�[38;5;28;01mFalse�[39;00m):

E               �[1;32m     91�[0m     �[38;5;124;03m"""�[39;00m

E               �[1;32m     92�[0m �[38;5;124;03m    Checks that the output schema is valid given the previous�[39;00m

E               �[1;32m     93�[0m �[38;5;124;03m    nodes in the graph and following nodes in the graph, as�[39;00m

E               �[0;32m   (...)�[0m

E               �[1;32m    107�[0m �[38;5;124;03m        If an output column is produced but not used by child nodes�[39;00m

E               �[1;32m    108�[0m �[38;5;124;03m    """�[39;00m

E               �[0;32m--> 109�[0m     �[38;5;28;43msuper�[39;49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mvalidate_schemas�[49m�[43m(�[49m�[43mroot_schema�[49m�[43m,�[49m�[43m �[49m�[43mstrict_dtypes�[49m�[43m)�[49m

E               �[1;32m    111�[0m     �[38;5;28;01mif�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mchildren:

E               �[1;32m    112�[0m         childrens_schema �[38;5;241m=�[39m Schema()

E

E               File �[0;32m~/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/node.py:226�[0m, in �[0;36mNode.validate_schemas�[0;34m(self, root_schema, strict_dtypes)�[0m

E               �[1;32m    224�[0m �[38;5;28;01mif�[39;00m strict_dtypes �[38;5;129;01mor�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mdynamic_dtypes:

E               �[1;32m    225�[0m     �[38;5;28;01mif�[39;00m source_col_schema�[38;5;241m.�[39mdtype �[38;5;241m!=�[39m col_schema�[38;5;241m.�[39mdtype:

E               �[0;32m--> 226�[0m         �[38;5;28;01mraise�[39;00m �[38;5;167;01mValueError�[39;00m(

E               �[1;32m    227�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mMismatched dtypes for column �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_name�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m provided to �[39m�[38;5;124m"�[39m

E               �[1;32m    228�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00m�[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39m�[38;5;18m__class__�[39m�[38;5;241m.�[39m�[38;5;18m__name__�[39m�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m: �[39m�[38;5;124m"�[39m

E               �[1;32m    229�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mancestor nodes provided dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00msource_col_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m, �[39m�[38;5;124m"�[39m

E               �[1;32m    230�[0m             �[38;5;124mf�[39m�[38;5;124m"�[39m�[38;5;124mexpected dtype �[39m�[38;5;124m'�[39m�[38;5;132;01m{�[39;00mcol_schema�[38;5;241m.�[39mdtype�[38;5;132;01m}�[39;00m�[38;5;124m'�[39m�[38;5;124m.�[39m�[38;5;124m"�[39m

E               �[1;32m    231�[0m         )

E               �[1;32m    233�[0m �[38;5;28mself�[39m�[38;5;241m.�[39mop�[38;5;241m.�[39mvalidate_schemas(

E               �[1;32m    234�[0m     parents_schema, deps_schema, �[38;5;28mself�[39m�[38;5;241m.�[39minput_schema, �[38;5;28mself�[39m�[38;5;241m.�[39moutput_schema, strict_dtypes

E               �[1;32m    235�[0m )

E

E               �[0;31mValueError�[0m: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.

E               ValueError: Mismatched dtypes for column 'user_id' provided to 'PredictImplicit': ancestor nodes provided dtype 'int64', expected dtype 'int32'.


../../../.local/lib/python3.8/site-packages/testbook/client.py:135: TestbookRuntimeError

_________________________ test_example_serving_xgboost _________________________


tb = <testbook.client.TestbookNotebookClient object at 0x7febbc339220>


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.notebook
@testbook(REPO_ROOT / "examples/Serving-An-XGboost-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_xgboost(tb):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """
    )
    NUM_OF_CELLS = len(tb.cells)
    # TODO: the following line is a hack -- remove when merlin-models#624 gets fixed
    tb.cells[4].source = tb.cells[4].source.replace(
        "without(['rating_binary', 'title'])", "without(['rating_binary', 'title', 'userId_count'])"
    )
    tb.execute_cell(list(range(0, 14)))





  with run_triton_server("ensemble", grpc_port=8001):





tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py:43:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = 'ensemble'


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

2022-10-19 15:27:13,508 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-10-19 15:27:13,514 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-10-19 15:27:13,516 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

2022-10-19 15:27:13,533 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize

[15:27:17] task [xgboost.dask-3]:tcp://127.0.0.1:44989 got new rank 0

I1019 15:27:19.217575 22620 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fbffe000000' with size 268435456

I1019 15:27:19.218407 22620 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1019 15:27:19.224113 22620 model_lifecycle.cc:459] loading: 1_predictforest:1

I1019 15:27:19.224156 22620 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1019 15:27:19.224204 22620 model_lifecycle.cc:459] loading: 1_fil:1

I1019 15:27:19.224244 22620 model_lifecycle.cc:459] loading: 1_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

1019 15:27:30.891525 22738 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictimplicit/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1019 15:27:30.948210 22620 model_lifecycle.cc:596] failed to load '1_predictimplicit' version 1: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictimplicit/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


I1019 15:27:30.964085 22620 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1019 15:27:30.964117 22620 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1019 15:27:30.964123 22620 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1019 15:27:30.964707 22620 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 1_predictforest (GPU device 0)

I1019 15:27:34.446089 22620 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1019 15:27:34.446457 22620 model_lifecycle.cc:693] successfully loaded '1_predictforest' version 1

I1019 15:27:37.407389 22620 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_fil (version 1)

I1019 15:27:37.408199 22620 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1019 15:27:37.408499 22620 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_fil_0 (GPU device 0)

I1019 15:27:37.432570 22620 model_lifecycle.cc:693] successfully loaded '1_fil' version 1

I1019 15:27:37.432843 22620 model_lifecycle.cc:459] loading: ensemble_model:1

I1019 15:27:37.432999 22620 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1019 15:27:37.433064 22620 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1019 15:27:37.433117 22620 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

| fil     | /opt/tritonserver/backends/fil/libtriton_fil.so       | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1019 15:27:37.433244 22620 server.cc:631]

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model               | Version | Status                                                                                                                                                                                                                                               |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_transformworkflow | 1       | READY                                                                                                                                                                                                                                                |

| 1_fil               | 1       | READY                                                                                                                                                                                                                                                |

| 1_predictforest     | 1       | READY                                                                                                                                                                                                                                                |

| 1_predictimplicit   | 1       | UNAVAILABLE: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py) |

|                     |         |                                                                                                                                                                                                                                                      |

|                     |         | At:                                                                                                                                                                                                                                                  |

|                     |         |   /var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictimplicit/1/model.py(35):                                                                                                                                              |

|                     |         |   (219): _call_with_frames_removed                                                                                                                                                                                      |

|                     |         |   (848): exec_module                                                                                                                                                                                           |

|                     |         |   (686): _load_unlocked                                                                                                                                                                                                 |

|                     |         |   (975): _find_and_load_unlocked                                                                                                                                                                                        |

|                     |         |   (991): _find_and_load                                                                                                                                                                                                 |

| ensemble_model      | 1       | READY                                                                                                                                                                                                                                                |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1019 15:27:37.490183 22620 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1019 15:27:37.491095 22620 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | ensemble                                                                                                                                                                                     |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1019 15:27:37.491132 22620 server.cc:262] Waiting for in-flight requests to complete.

I1019 15:27:37.491142 22620 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1019 15:27:37.491302 22620 server.cc:293] All models are stopped, unloading models

I1019 15:27:37.491314 22620 server.cc:300] Timeout 30: Found 4 live models and 0 in-flight non-inference requests

I1019 15:27:37.491341 22620 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1019 15:27:37.491404 22620 instance_finalize.hpp:36] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1019 15:27:37.491545 22620 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state

I1019 15:27:37.491583 22620 model_lifecycle.cc:578] successfully unloaded '1_fil' version 1

I1019 15:27:38.491416 22620 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1019 15:27:38.519692 22620 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(

I1019 15:27:39.070182 22620 model_lifecycle.cc:578] successfully unloaded '1_predictforest' version 1

I1019 15:27:39.129451 22620 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1019 15:27:39.491541 22620 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1019 15:27:39.519881 22620 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 39 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              71     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2287    885    61%


============ 3 failed, 84 passed, 56 warnings in 507.81s (0:08:27) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins16994379963456562173.sh

@karlhigley
Copy link
Contributor Author

karlhigley commented Oct 19, 2022

Tests should pass once we merge NVIDIA-Merlin/core#153 (small fixes to handle list dtypes)

@oliverholworthy
Copy link
Contributor

oliverholworthy commented Oct 19, 2022

| 1_predictimplicit | 1 | UNAVAILABLE: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py) |

I think the reason we're getting this error may be due to the example notebook tests not writing the data out to a tmpdir. So it's using a model repository directory from a previous run (which still had the InferenceDataFrame).

We might need to update that to write out to a temporary directory

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 20, 2022

Click to view CI Results
GitHub pull request #204 of commit 36a4fe0c091ded2cd48e18a113c9437fb2d4ff07, no merge conflicts.
Running as SYSTEM
Setting status of 36a4fe0c091ded2cd48e18a113c9437fb2d4ff07 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/439/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 36a4fe0c091ded2cd48e18a113c9437fb2d4ff07^{commit} # timeout=10
Checking out Revision 36a4fe0c091ded2cd48e18a113c9437fb2d4ff07 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 36a4fe0c091ded2cd48e18a113c9437fb2d4ff07 # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk 63b01ccc9f10bbff666bb399fe6dcd607f63907c # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins14806424101303255795.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+16.g36a4fe0.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.95,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.94,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+16.g36a4fe0,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3640549287'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-fsvxr6xr
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-fsvxr6xr
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c405f4719782ee057b29ef2b069731fffb096a50
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (21.3)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (4.64.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (7.0.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (1.3.5)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (0.55.1)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (3.1.2)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+11.gc405f47) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+11.gc405f47-py3-none-any.whl size=118282 sha256=5ec1e9cde510a91258569e297c933e44bfb494f97a86ad897a69667c013c5e15
  Stored in directory: /tmp/pip-ephem-wheel-cache-onvy24jm/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+11.gc405f47

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+16.g36a4fe0) (0.7.0+11.gc405f47)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+16.g36a4fe0) (1.1.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (21.3)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.10.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2022.5.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (4.64.1)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (7.0.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.2.5)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.3.5)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2022.3.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (0.55.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (3.19.5)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.2.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (5.4.1)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (0.12.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2.4.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (8.1.3)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (3.1.2)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.0.4)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (6.1)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2.0.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (5.8.0)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (65.3.0)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+16.g36a4fe0) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+16.g36a4fe0-py3-none-any.whl size=83566 sha256=0a9df61b7c3584fce26054e85c18a7d18b478d19290213be7e53e100bd6867d0

Stored in directory: /tmp/pip-ephem-wheel-cache-3n4foj26/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+16.g36a4fe0

Uninstalling merlin-systems-0.6.0+16.g36a4fe0:

Successfully uninstalled merlin-systems-0.6.0+16.g36a4fe0

Successfully installed merlin-systems-0.6.0+16.g36a4fe0


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 86 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py F [  2%]

F                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ....                            [ 11%]

tests/unit/systems/dag/test_graph.py ..                                  [ 13%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 16%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 20%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 23%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 24%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 32%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 33%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 69%]

tests/unit/systems/ops/implicit/test_op.py .....F...                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_____________________ test_example_serving_implicit[True] ______________________


tb = <testbook.client.TestbookNotebookClient object at 0x7f680179c3d0>

gpu = True


@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
@testbook(REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_implicit(tb, gpu):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """,
        pop=True,
    )
    NUM_OF_CELLS = len(tb.cells)
    if not gpu:
        tb.cells[
            6
        ].source = (
            "model = BayesianPersonalizedRanking(use_gpu=False)\nmodel.fit(train_transformed)"
        )
    tb.execute_cell(list(range(0, 16)))





  with run_triton_server("ensemble", grpc_port=8001):





tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:54:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = 'ensemble'


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1020 14:26:56.535632 13055 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f1ab6000000' with size 268435456

I1020 14:26:56.536419 13055 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1020 14:26:56.542067 13055 model_lifecycle.cc:459] loading: 1_fil:1

I1020 14:26:56.542104 13055 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1020 14:26:56.542133 13055 model_lifecycle.cc:459] loading: 1_predictforest:1

I1020 14:26:56.542179 13055 model_lifecycle.cc:459] loading: 1_predictimplicit:1

1020 14:26:59.416956 13094 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1020 14:26:59.481315 13055 model_lifecycle.cc:596] failed to load '1_predictforest' version 1: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


I1020 14:27:03.859251 13055 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1020 14:27:03.859290 13055 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1020 14:27:03.859297 13055 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1020 14:27:03.859851 13055 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1020 14:27:06.848783 13055 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_fil (version 1)

I1020 14:27:06.849152 13055 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 14:27:11.216145 13055 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_fil_0 (GPU device 0)

I1020 14:27:11.237685 13055 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 1_predictimplicit (GPU device 0)

I1020 14:27:11.239332 13055 model_lifecycle.cc:693] successfully loaded '1_fil' version 1

I1020 14:27:14.872136 13055 model_lifecycle.cc:693] successfully loaded '1_predictimplicit' version 1

I1020 14:27:14.872516 13055 model_lifecycle.cc:459] loading: ensemble_model:1

I1020 14:27:14.872857 13055 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1020 14:27:14.872979 13055 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1020 14:27:14.873071 13055 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

| fil     | /opt/tritonserver/backends/fil/libtriton_fil.so       | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:27:14.873299 13055 server.cc:631]

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model               | Version | Status                                                                                                                                                                                                                                               |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_transformworkflow | 1       | READY                                                                                                                                                                                                                                                |

| 1_fil               | 1       | READY                                                                                                                                                                                                                                                |

| 1_predictforest     | 1       | UNAVAILABLE: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py) |

|                     |         |                                                                                                                                                                                                                                                      |

|                     |         | At:                                                                                                                                                                                                                                                  |

|                     |         |   /var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35):                                                                                                                                                |

|                     |         |   (219): _call_with_frames_removed                                                                                                                                                                                      |

|                     |         |   (848): exec_module                                                                                                                                                                                           |

|                     |         |   (686): _load_unlocked                                                                                                                                                                                                 |

|                     |         |   (975): _find_and_load_unlocked                                                                                                                                                                                        |

|                     |         |   (991): _find_and_load                                                                                                                                                                                                 |

| 1_predictimplicit   | 1       | READY                                                                                                                                                                                                                                                |

| ensemble_model      | 1       | READY                                                                                                                                                                                                                                                |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:27:14.936460 13055 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1020 14:27:14.937348 13055 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | ensemble                                                                                                                                                                                     |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:27:14.937384 13055 server.cc:262] Waiting for in-flight requests to complete.

I1020 14:27:14.937394 13055 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1020 14:27:14.937585 13055 server.cc:293] All models are stopped, unloading models

I1020 14:27:14.937589 13055 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1020 14:27:14.937596 13055 server.cc:300] Timeout 30: Found 4 live models and 0 in-flight non-inference requests

I1020 14:27:14.937719 13055 instance_finalize.hpp:36] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1020 14:27:14.937940 13055 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state

I1020 14:27:14.937995 13055 model_lifecycle.cc:578] successfully unloaded '1_fil' version 1

I1020 14:27:15.937698 13055 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1020 14:27:15.964003 13055 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 14:27:16.403292 13055 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1020 14:27:16.555724 13055 model_lifecycle.cc:578] successfully unloaded '1_predictimplicit' version 1

I1020 14:27:16.937827 13055 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1020 14:27:16.964199 13055 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

_____________________ test_example_serving_implicit[False] _____________________


tb = <testbook.client.TestbookNotebookClient object at 0x7f680179c3d0>

gpu = False


@pytest.mark.notebook
@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("gpu", _TRAIN_ON_GPU)
@testbook(REPO_ROOT / "examples/Serving-An-Implicit-Model-With-Merlin-Systems.ipynb", execute=False)
def test_example_serving_implicit(tb, gpu):
    tb.inject(
        """
        from unittest.mock import patch
        from merlin.datasets.synthetic import generate_data
        mock_train, mock_valid = generate_data(
            input="movielens-100k",
            num_rows=1000,
            set_sizes=(0.8, 0.2)
        )
        p1 = patch(
            "merlin.datasets.entertainment.get_movielens",
            return_value=[mock_train, mock_valid]
        )
        p1.start()
        """,
        pop=True,
    )
    NUM_OF_CELLS = len(tb.cells)
    if not gpu:
        tb.cells[
            6
        ].source = (
            "model = BayesianPersonalizedRanking(use_gpu=False)\nmodel.fit(train_transformed)"
        )
    tb.execute_cell(list(range(0, 16)))





  with run_triton_server("ensemble", grpc_port=8001):





tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py:54:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = 'ensemble'


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------

I1020 14:27:26.232015 13473 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f8e82000000' with size 268435456

I1020 14:27:26.232772 13473 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1020 14:27:26.238078 13473 model_lifecycle.cc:459] loading: 1_fil:1

I1020 14:27:26.238114 13473 model_lifecycle.cc:459] loading: 0_transformworkflow:1

I1020 14:27:26.238139 13473 model_lifecycle.cc:459] loading: 1_predictforest:1

I1020 14:27:26.238223 13473 model_lifecycle.cc:459] loading: 1_predictimplicit:1

I1020 14:27:26.258444 13473 initialize.hpp:43] TRITONBACKEND_Initialize: fil

I1020 14:27:26.258470 13473 backend.hpp:47] Triton TRITONBACKEND API version: 1.10

I1020 14:27:26.258477 13473 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10

I1020 14:27:26.259058 13473 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: 1_fil (version 1)

I1020 14:27:26.263266 13473 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: 1_fil_0 (GPU device 0)

I1020 14:27:26.313578 13473 model_lifecycle.cc:693] successfully loaded '1_fil' version 1

1020 14:27:29.146804 13528 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1020 14:27:29.202827 13473 model_lifecycle.cc:596] failed to load '1_predictforest' version 1: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py)


At:

/var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 14:27:37.959212 13473 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_transformworkflow (GPU device 0)

I1020 14:27:40.890573 13473 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 1_predictimplicit (GPU device 0)

I1020 14:27:40.890902 13473 model_lifecycle.cc:693] successfully loaded '0_transformworkflow' version 1

I1020 14:27:43.823303 13473 model_lifecycle.cc:693] successfully loaded '1_predictimplicit' version 1

I1020 14:27:43.823641 13473 model_lifecycle.cc:459] loading: ensemble_model:1

I1020 14:27:43.823986 13473 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1020 14:27:43.824168 13473 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1020 14:27:43.824268 13473 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| fil     | /opt/tritonserver/backends/fil/libtriton_fil.so       | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:27:43.824480 13473 server.cc:631]

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model               | Version | Status                                                                                                                                                                                                                                               |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_transformworkflow | 1       | READY                                                                                                                                                                                                                                                |

| 1_fil               | 1       | READY                                                                                                                                                                                                                                                |

| 1_predictforest     | 1       | UNAVAILABLE: Internal: ImportError: cannot import name 'InferenceDataFrame' from 'merlin.systems.dag.ops.operator' (/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/operator.py) |

|                     |         |                                                                                                                                                                                                                                                      |

|                     |         | At:                                                                                                                                                                                                                                                  |

|                     |         |   /var/jenkins_home/workspace/merlin_systems/systems/ensemble/1_predictforest/1/model.py(35):                                                                                                                                                |

|                     |         |   (219): _call_with_frames_removed                                                                                                                                                                                      |

|                     |         |   (848): exec_module                                                                                                                                                                                           |

|                     |         |   (686): _load_unlocked                                                                                                                                                                                                 |

|                     |         |   (975): _find_and_load_unlocked                                                                                                                                                                                        |

|                     |         |   (991): _find_and_load                                                                                                                                                                                                 |

| 1_predictimplicit   | 1       | READY                                                                                                                                                                                                                                                |

| ensemble_model      | 1       | READY                                                                                                                                                                                                                                                |

+---------------------+---------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:27:43.888756 13473 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1020 14:27:43.889610 13473 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | ensemble                                                                                                                                                                                     |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:27:43.889647 13473 server.cc:262] Waiting for in-flight requests to complete.

I1020 14:27:43.889659 13473 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1020 14:27:43.889849 13473 server.cc:293] All models are stopped, unloading models

I1020 14:27:43.889865 13473 server.cc:300] Timeout 30: Found 4 live models and 0 in-flight non-inference requests

I1020 14:27:43.889873 13473 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1020 14:27:43.889997 13473 instance_finalize.hpp:36] TRITONBACKEND_ModelInstanceFinalize: delete instance state

I1020 14:27:43.890216 13473 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state

I1020 14:27:43.890276 13473 model_lifecycle.cc:578] successfully unloaded '1_fil' version 1

I1020 14:27:44.889956 13473 server.cc:300] Timeout 29: Found 2 live models and 0 in-flight non-inference requests

W1020 14:27:44.915110 13473 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 14:27:45.305498 13473 model_lifecycle.cc:578] successfully unloaded '1_predictimplicit' version 1

I1020 14:27:45.357393 13473 model_lifecycle.cc:578] successfully unloaded '0_transformworkflow' version 1

I1020 14:27:45.890086 13473 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1020 14:27:45.915317 13473 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

_______________ test_ensemble[AlternatingLeastSquares-ensemble] ________________


model_cls = <function AlternatingLeastSquares at 0x7f67ffb5caf0>

backend = 'ensemble'

tmpdir = local('/tmp/pytest-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("backend", ["ensemble", "executor"])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, backend, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema, name=f"{backend}_model")
    triton_ens.export(tmpdir, backend=backend)

    model_name = triton_ens.name
    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None

    with run_triton_server(tmpdir) as client:





      response = client.infer(model_name, inputs, outputs=outputs)





tests/unit/systems/ops/implicit/test_op.py:122:




/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322: in infer

raise_error_grpc(rpc_error)




rpc_error = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "in ensemble 'ensemble_model'...-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute\n","grpc_status":13}"






def raise_error_grpc(rpc_error):





  raise get_error_grpc(rpc_error) from None





E       tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '0_predictimplicit', message: TypeError: init(): incompatible constructor arguments. The following argument types are supported:

E           1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)

E

E       Invoked with: kwargs: tensors=[], error="<class 'RuntimeError'>, cublas error: CUBLAS_STATUS_NOT_INITIALIZED (/project/implicit/gpu/knn.cu:87), [<FrameSummary file /tmp/pytest-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py, line 105 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py, line 59 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py, line 176 in transform>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 87 in recommend>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 122 in knn>, <FrameSummary file _cuda.pyx, line 47 in implicit.gpu._cuda.KnnQuery.cinit>]"

E

E       At:

E         /tmp/pytest-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute


/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62: InferenceServerException

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2651.71it/s]

I1020 14:32:25.485779 17207 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f9724000000' with size 268435456

I1020 14:32:25.486558 17207 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1020 14:32:25.490012 17207 model_lifecycle.cc:459] loading: 0_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 14:32:29.860083 17207 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_predictimplicit (GPU device 0)

I1020 14:32:33.698565 17207 model_lifecycle.cc:693] successfully loaded '0_predictimplicit' version 1

I1020 14:32:33.698813 17207 model_lifecycle.cc:459] loading: ensemble_model:1

I1020 14:32:33.698969 17207 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1020 14:32:33.699040 17207 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1020 14:32:33.699116 17207 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:32:33.699172 17207 server.cc:631]

+-------------------+---------+--------+

| Model             | Version | Status |

+-------------------+---------+--------+

| 0_predictimplicit | 1       | READY  |

| ensemble_model    | 1       | READY  |

+-------------------+---------+--------+


I1020 14:32:33.763633 17207 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1020 14:32:33.764500 17207 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:32:33.765925 17207 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1020 14:32:33.766131 17207 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1020 14:32:33.807049 17207 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1020 14:32:34.788578 17207 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1020 14:32:35.788781 17207 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

1020 14:32:36.810377 17286 pb_stub.cc:777] Failed to process the request(s) for model '0_predictimplicit', message: TypeError: init(): incompatible constructor arguments. The following argument types are supported:

1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)


Invoked with: kwargs: tensors=[], error="<class 'RuntimeError'>, cublas error: CUBLAS_STATUS_NOT_INITIALIZED (/project/implicit/gpu/knn.cu:87), [<FrameSummary file /tmp/pytest-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py, line 105 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py, line 59 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py, line 176 in transform>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 87 in recommend>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 122 in knn>, <FrameSummary file _cuda.pyx, line 47 in implicit.gpu._cuda.KnnQuery.cinit>]"


At:

/tmp/pytest-of-jenkins/pytest-4/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute


I1020 14:32:36.811765 17207 server.cc:262] Waiting for in-flight requests to complete.

I1020 14:32:36.811779 17207 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1020 14:32:36.811901 17207 server.cc:293] All models are stopped, unloading models

I1020 14:32:36.811911 17207 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1020 14:32:36.812019 17207 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

W1020 14:32:36.813371 17207 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1020 14:32:37.811996 17207 server.cc:300] Timeout 29: Found 1 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(

I1020 14:32:38.392058 17207 model_lifecycle.cc:578] successfully unloaded '0_predictimplicit' version 1

I1020 14:32:38.812140 17207 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              72     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2288    885    61%


============ 3 failed, 83 passed, 56 warnings in 549.79s (0:09:09) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins997371795700344757.sh

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 20, 2022

Click to view CI Results
GitHub pull request #204 of commit fa7f55867618024b47c59ab46c20f7ed2567e3c4, no merge conflicts.
Running as SYSTEM
Setting status of fa7f55867618024b47c59ab46c20f7ed2567e3c4 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/440/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse fa7f55867618024b47c59ab46c20f7ed2567e3c4^{commit} # timeout=10
Checking out Revision fa7f55867618024b47c59ab46c20f7ed2567e3c4 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f fa7f55867618024b47c59ab46c20f7ed2567e3c4 # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk 36a4fe0c091ded2cd48e18a113c9437fb2d4ff07 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins10124958459327332970.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+18.gfa7f558.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.95,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.94,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+18.gfa7f558,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='4136074436'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-w5ruxf9p
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-w5ruxf9p
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c405f4719782ee057b29ef2b069731fffb096a50
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (21.3)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (0.55.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.4.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.0.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (3.1.2)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.8.0)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+11.gc405f47) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+11.gc405f47-py3-none-any.whl size=118282 sha256=c0ab2790721b764c36625f86932a5f2ef1827636698aab9b7c52ff28702fca4d
  Stored in directory: /tmp/pip-ephem-wheel-cache-_2a5vdzb/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+11.gc405f47

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+18.gfa7f558) (0.7.0+11.gc405f47)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+18.gfa7f558) (1.1.1)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.6.0+18.gfa7f558) (2.22.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (3.19.5)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (7.0.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.3.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (21.3)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (0.55.1)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.10.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2022.5.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2022.3.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2022.3.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (4.64.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2.2.0)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (5.4.1)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2.0.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.7.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2.4.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (3.1.2)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (6.1)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.0.4)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (8.1.3)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (5.8.0)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (65.3.0)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (3.0.9)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2022.2.1)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2.8.2)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+18.gfa7f558) (6.0.1)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+18.gfa7f558-py3-none-any.whl size=83600 sha256=0aae1d70803df2fdeb435e904a4d8aecb2a6981b562e6a4bf2bb63593215cf9d

Stored in directory: /tmp/pip-ephem-wheel-cache-0qxd1qu5/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+18.gfa7f558

Uninstalling merlin-systems-0.6.0+18.gfa7f558:

Successfully uninstalled merlin-systems-0.6.0+18.gfa7f558

Successfully installed merlin-systems-0.6.0+18.gfa7f558


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 86 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ....                            [ 11%]

tests/unit/systems/dag/test_graph.py ..                                  [ 13%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 16%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 20%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 23%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 24%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 32%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 33%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 69%]

tests/unit/systems/ops/implicit/test_op.py .....F...                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_______________ test_ensemble[AlternatingLeastSquares-ensemble] ________________


model_cls = <function AlternatingLeastSquares at 0x7f3901939af0>

backend = 'ensemble'

tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("backend", ["ensemble", "executor"])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, backend, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema, name=f"{backend}_model")
    triton_ens.export(tmpdir, backend=backend)

    model_name = triton_ens.name
    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None

    with run_triton_server(tmpdir) as client:





      response = client.infer(model_name, inputs, outputs=outputs)





tests/unit/systems/ops/implicit/test_op.py:122:




/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322: in infer

raise_error_grpc(rpc_error)




rpc_error = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "in ensemble 'ensemble_model'...-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute\n","grpc_status":13}"






def raise_error_grpc(rpc_error):





  raise get_error_grpc(rpc_error) from None





E       tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '0_predictimplicit', message: TypeError: init(): incompatible constructor arguments. The following argument types are supported:

E           1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)

E

E       Invoked with: kwargs: tensors=[], error="<class 'RuntimeError'>, cublas error: CUBLAS_STATUS_NOT_INITIALIZED (/project/implicit/gpu/knn.cu:87), [<FrameSummary file /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py, line 105 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py, line 59 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py, line 176 in transform>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 87 in recommend>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 122 in knn>, <FrameSummary file _cuda.pyx, line 47 in implicit.gpu._cuda.KnnQuery.cinit>]"

E

E       At:

E         /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute


/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62: InferenceServerException

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2399.67it/s]

I1020 14:42:55.346425 24641 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fdfa6000000' with size 268435456

I1020 14:42:55.347154 24641 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1020 14:42:55.350494 24641 model_lifecycle.cc:459] loading: 0_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 14:42:59.744688 24641 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_predictimplicit (GPU device 0)

I1020 14:43:03.502597 24641 model_lifecycle.cc:693] successfully loaded '0_predictimplicit' version 1

I1020 14:43:03.502794 24641 model_lifecycle.cc:459] loading: ensemble_model:1

I1020 14:43:03.503076 24641 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1020 14:43:03.503158 24641 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1020 14:43:03.503211 24641 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:43:03.503255 24641 server.cc:631]

+-------------------+---------+--------+

| Model             | Version | Status |

+-------------------+---------+--------+

| 0_predictimplicit | 1       | READY  |

| ensemble_model    | 1       | READY  |

+-------------------+---------+--------+


I1020 14:43:03.561743 24641 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1020 14:43:03.562561 24641 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 14:43:03.564073 24641 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1020 14:43:03.564267 24641 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1020 14:43:03.605154 24641 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1020 14:43:04.588734 24641 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1020 14:43:05.588945 24641 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

W1020 14:43:06.606971 24641 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

1020 14:43:11.707515 24720 pb_stub.cc:777] Failed to process the request(s) for model '0_predictimplicit', message: TypeError: init(): incompatible constructor arguments. The following argument types are supported:

1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)


Invoked with: kwargs: tensors=[], error="<class 'RuntimeError'>, cublas error: CUBLAS_STATUS_NOT_INITIALIZED (/project/implicit/gpu/knn.cu:87), [<FrameSummary file /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py, line 105 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py, line 59 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py, line 176 in transform>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 87 in recommend>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 122 in knn>, <FrameSummary file _cuda.pyx, line 47 in implicit.gpu._cuda.KnnQuery.cinit>]"


At:

/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute


I1020 14:43:11.709025 24641 server.cc:262] Waiting for in-flight requests to complete.

I1020 14:43:11.709038 24641 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1020 14:43:11.709140 24641 server.cc:293] All models are stopped, unloading models

I1020 14:43:11.709151 24641 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1020 14:43:11.709221 24641 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

I1020 14:43:12.709227 24641 server.cc:300] Timeout 29: Found 1 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(

I1020 14:43:13.061086 24641 model_lifecycle.cc:578] successfully unloaded '0_predictimplicit' version 1

I1020 14:43:13.709348 24641 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              72     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     14    81%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2288    887    61%


============ 1 failed, 85 passed, 56 warnings in 586.47s (0:09:46) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins6589882792018631744.sh

@oliverholworthy oliverholworthy mentioned this pull request Oct 20, 2022
Merged
oliverholworthy
oliverholworthy approved these changes Oct 20, 2022
View reviewed changes
Copy link
Contributor

@oliverholworthy oliverholworthy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's merge once the tests are passing. #214 fixes the issue with the examples

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 20, 2022

Click to view CI Results
GitHub pull request #204 of commit 521f44a107abd162339805d06faff8fa3620efdd, no merge conflicts.
Running as SYSTEM
Setting status of 521f44a107abd162339805d06faff8fa3620efdd to PENDING with url https://10.20.13.93:8080/job/merlin_systems/447/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 521f44a107abd162339805d06faff8fa3620efdd^{commit} # timeout=10
Checking out Revision 521f44a107abd162339805d06faff8fa3620efdd (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk 2a83be95889d630455d400abfddeb32b1b312750 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins9749930384141402846.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+20.g521f44a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.95,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.94,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+20.g521f44a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1381322638'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-bif9ow6k
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-bif9ow6k
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c405f4719782ee057b29ef2b069731fffb096a50
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (21.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.2.5)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.10.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (4.64.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.5.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (7.0.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (0.55.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.4)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (3.1.2)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (65.3.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+11.gc405f47) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+11.gc405f47-py3-none-any.whl size=118282 sha256=f9d6c532848fa102091dd2ac9037698aaf27447851c24d21dc0e171edb6c48a1
  Stored in directory: /tmp/pip-ephem-wheel-cache-05eyo_bl/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+11.gc405f47

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.6.0+20.g521f44a) (2.22.0)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (1.1.1)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (0.7.0+11.gc405f47)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (21.3)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.5)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.10.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.64.1)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.5.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.19.5)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (7.0.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.3.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.55.1)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.4.3)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.12.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.4.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.8.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.7.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.4)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.1)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.0.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (8.1.3)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.1.2)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (65.3.0)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.1)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.2)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.1.0)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+20.g521f44a-py3-none-any.whl size=83592 sha256=49f768c908a85296ade63eac923f5bbe269c65f7e65ac91a7314a46de6a3bb5f

Stored in directory: /tmp/pip-ephem-wheel-cache-yiic93l5/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+20.g521f44a

Uninstalling merlin-systems-0.6.0+20.g521f44a:

Successfully uninstalled merlin-systems-0.6.0+20.g521f44a

Successfully installed merlin-systems-0.6.0+20.g521f44a


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 86 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ....                            [ 11%]

tests/unit/systems/dag/test_graph.py ..                                  [ 13%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 16%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 20%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 23%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 24%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 32%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 33%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 69%]

tests/unit/systems/ops/implicit/test_op.py .....F...                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_______________ test_ensemble[AlternatingLeastSquares-ensemble] ________________


model_cls = <function AlternatingLeastSquares at 0x7f2106042a60>

backend = 'ensemble'

tmpdir = local('/tmp/pytest-of-jenkins/pytest-7/test_ensemble_AlternatingLeast0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("backend", ["ensemble", "executor"])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, backend, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema, name=f"{backend}_model")
    triton_ens.export(tmpdir, backend=backend)

    model_name = triton_ens.name
    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/implicit/test_op.py:121:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-7/test_ensemble_AlternatingLeast0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2395.65it/s]

I1020 18:43:34.568119 7283 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7fe596000000' with size 268435456

I1020 18:43:34.568853 7283 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1020 18:43:34.572208 7283 model_lifecycle.cc:459] loading: 0_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 18:43:38.963598 7283 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_predictimplicit (GPU device 0)

1020 18:43:42.321738 7403 pb_stub.cc:309] Failed to initialize Python stub: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory


At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu

/usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init

/tmp/pytest-of-jenkins/pytest-7/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize


/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(

E1020 18:43:42.769415 7283 model_lifecycle.cc:596] failed to load '0_predictimplicit' version 1: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory


At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu

/usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init

/tmp/pytest-of-jenkins/pytest-7/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize


E1020 18:43:42.769604 7283 model_repository_manager.cc:481] Invalid argument: ensemble 'ensemble_model' depends on '0_predictimplicit' which has no loaded version

I1020 18:43:42.769704 7283 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1020 18:43:42.769799 7283 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 18:43:42.769971 7283 server.cc:631]

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model             | Version | Status                                                                                                                                                                                                                                    |

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predictimplicit | 1       | UNAVAILABLE: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory |

|                   |         |                                                                                                                                                                                                                                           |

|                   |         | At:                                                                                                                                                                                                                                       |

|                   |         |   _cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit                                                                                                                                                                                     |

|                   |         |   /usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu                                                                                                                                                                 |

|                   |         |   /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load                                                                                                                                             |

|                   |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config                                                                                       |

|                   |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init                                                                                              |

|                   |         |   /tmp/pytest-of-jenkins/pytest-7/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize                                                                                                                            |

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 18:43:42.836243 7283 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1020 18:43:42.837149 7283 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-7/test_ensemble_AlternatingLeast0                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 18:43:42.837162 7283 server.cc:262] Waiting for in-flight requests to complete.

I1020 18:43:42.837169 7283 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1020 18:43:42.837187 7283 server.cc:293] All models are stopped, unloading models

I1020 18:43:42.837193 7283 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1020 18:43:43.855906 7283 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              72     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2288    885    61%


============ 1 failed, 85 passed, 56 warnings in 578.94s (0:09:38) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins3474296694046110517.sh

@karlhigley
Copy link
Contributor Author

karlhigley commented Oct 20, 2022

No idea what's going on with the Implicit memory allocation failure here 😓

@jperez999
Copy link
Collaborator

jperez999 commented Oct 20, 2022

rerun tests

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 20, 2022

Click to view CI Results
GitHub pull request #204 of commit 521f44a107abd162339805d06faff8fa3620efdd, no merge conflicts.
GitHub pull request #204 of commit 521f44a107abd162339805d06faff8fa3620efdd, no merge conflicts.
Running as SYSTEM
Setting status of 521f44a107abd162339805d06faff8fa3620efdd to PENDING with url https://10.20.13.93:8080/job/merlin_systems/448/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 521f44a107abd162339805d06faff8fa3620efdd^{commit} # timeout=10
Checking out Revision 521f44a107abd162339805d06faff8fa3620efdd (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins11352840701161889264.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+20.g521f44a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.95,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.94,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+20.g521f44a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3872018275'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-d1lva8yp
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-d1lva8yp
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c405f4719782ee057b29ef2b069731fffb096a50
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (3.19.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (7.0.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.2.5)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.12.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (3.1.2)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (6.1)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (65.3.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+11.gc405f47) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+11.gc405f47-py3-none-any.whl size=118282 sha256=d7a75f3212cad4265dc49af10ad1ace7b597f75eb5ec234897ca975e7059b2a0
  Stored in directory: /tmp/pip-ephem-wheel-cache-j4vqbbk4/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+11.gc405f47

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (0.7.0+11.gc405f47)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.6.0+20.g521f44a) (2.22.0)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (1.1.1)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.19.5)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (7.0.0)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.3.5)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.5.0)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.55.1)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (21.3)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.64.1)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.10.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.5)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.4.3)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.12.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.1.2)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.4.0)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.0.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (8.1.3)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.1)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.7.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.4)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.8.0)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.38.1)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.20.3)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (65.3.0)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.1)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+20.g521f44a-py3-none-any.whl size=83592 sha256=131992d7f0b3cacf34b05d3a4c0b5939881e13f65075732ce75898d4fbd9b717

Stored in directory: /tmp/pip-ephem-wheel-cache-2wiw9h9r/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+20.g521f44a

Uninstalling merlin-systems-0.6.0+20.g521f44a:

Successfully uninstalled merlin-systems-0.6.0+20.g521f44a

Successfully installed merlin-systems-0.6.0+20.g521f44a


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 86 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ....                            [ 11%]

tests/unit/systems/dag/test_graph.py ..                                  [ 13%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 16%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 20%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 23%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 24%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 32%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 33%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 69%]

tests/unit/systems/ops/implicit/test_op.py .....F...                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_______________ test_ensemble[AlternatingLeastSquares-ensemble] ________________


model_cls = <function AlternatingLeastSquares at 0x7f688a5dba60>

backend = 'ensemble'

tmpdir = local('/tmp/pytest-of-jenkins/pytest-11/test_ensemble_AlternatingLeast0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("backend", ["ensemble", "executor"])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, backend, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema, name=f"{backend}_model")
    triton_ens.export(tmpdir, backend=backend)

    model_name = triton_ens.name
    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/implicit/test_op.py:121:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-11/test_ensemble_AlternatingLeast0')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:81: RuntimeError

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2912.71it/s]

I1020 20:02:36.306745 31448 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f3a66000000' with size 268435456

I1020 20:02:36.307485 31448 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1020 20:02:36.310842 31448 model_lifecycle.cc:459] loading: 0_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1020 20:02:40.719288 31448 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_predictimplicit (GPU device 0)

1020 20:02:44.085488 31602 pb_stub.cc:309] Failed to initialize Python stub: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory


At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu

/usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init

/tmp/pytest-of-jenkins/pytest-11/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize


/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(

E1020 20:02:44.511437 31448 model_lifecycle.cc:596] failed to load '0_predictimplicit' version 1: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory


At:

_cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit

/usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu

/usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init

/tmp/pytest-of-jenkins/pytest-11/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize


E1020 20:02:44.511545 31448 model_repository_manager.cc:481] Invalid argument: ensemble 'ensemble_model' depends on '0_predictimplicit' which has no loaded version

I1020 20:02:44.511595 31448 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1020 20:02:44.511631 31448 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 20:02:44.511710 31448 server.cc:631]

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model             | Version | Status                                                                                                                                                                                                                                    |

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| 0_predictimplicit | 1       | UNAVAILABLE: Internal: MemoryError: std::bad_alloc: out_of_memory: CUDA error at: /project/_skbuild/linux-x86_64-3.8/cmake-build/_deps/rmm-src/include/rmm/mr/device/cuda_memory_resource.hpp:70: cudaErrorMemoryAllocation out of memory |

|                   |         |                                                                                                                                                                                                                                           |

|                   |         | At:                                                                                                                                                                                                                                       |

|                   |         |   _cuda.pyx(104): implicit.gpu._cuda.Matrix.cinit                                                                                                                                                                                     |

|                   |         |   /usr/local/lib/python3.8/dist-packages/implicit/cpu/als.py(441): to_gpu                                                                                                                                                                 |

|                   |         |   /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py(212): load                                                                                                                                             |

|                   |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py(154): from_config                                                                                       |

|                   |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py(45): init                                                                                              |

|                   |         |   /tmp/pytest-of-jenkins/pytest-11/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(61): initialize                                                                                                                           |

+-------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 20:02:44.572641 31448 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1020 20:02:44.573614 31448 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-11/test_ensemble_AlternatingLeast0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1020 20:02:44.573629 31448 server.cc:262] Waiting for in-flight requests to complete.

I1020 20:02:44.573636 31448 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1020 20:02:44.573654 31448 server.cc:293] All models are stopped, unloading models

I1020 20:02:44.573660 31448 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1020 20:02:45.590243 31448 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              72     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     12    83%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2288    885    61%


============ 1 failed, 85 passed, 56 warnings in 569.49s (0:09:29) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins17293611824856154336.sh

@jperez999
Copy link
Collaborator

jperez999 commented Oct 21, 2022

rerun tests

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 21, 2022

Click to view CI Results
GitHub pull request #204 of commit 521f44a107abd162339805d06faff8fa3620efdd, no merge conflicts.
GitHub pull request #204 of commit 521f44a107abd162339805d06faff8fa3620efdd, no merge conflicts.
Running as SYSTEM
Setting status of 521f44a107abd162339805d06faff8fa3620efdd to PENDING with url https://10.20.13.93:8080/job/merlin_systems/449/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 521f44a107abd162339805d06faff8fa3620efdd^{commit} # timeout=10
Checking out Revision 521f44a107abd162339805d06faff8fa3620efdd (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins12181564529541762379.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+20.g521f44a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.95,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.94,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+20.g521f44a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='4075711154'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-93w9oevl
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-93w9oevl
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c405f4719782ee057b29ef2b069731fffb096a50
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (4.64.1)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (0.4.3)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (6.1)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (8.1.3)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.7.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.4.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+11.gc405f47) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+11.gc405f47-py3-none-any.whl size=118282 sha256=e4f133f09a32eceed2f20c1e575d4a3dbf076707c0b7dbb43588e68db601d0aa
  Stored in directory: /tmp/pip-ephem-wheel-cache-4bsei6ch/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+11.gc405f47

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (1.1.1)

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (0.7.0+11.gc405f47)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.6.0+20.g521f44a) (2.22.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (7.0.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.5)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.19.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (21.3)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.3.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.55.1)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.10.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.5.0)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.64.1)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.4.3)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.12.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.4.1)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.2.0)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.1)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (8.1.3)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.4)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.0.0)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.8.0)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.7.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.4.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.1.2)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (65.3.0)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.1.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.0.0)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.1)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+20.g521f44a-py3-none-any.whl size=83592 sha256=c4fa80b0d3d5cbc242740e58bf32c41a2f5eff177de7c4d7784c5dbc892879d6

Stored in directory: /tmp/pip-ephem-wheel-cache-shc63zdc/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+20.g521f44a

Uninstalling merlin-systems-0.6.0+20.g521f44a:

Successfully uninstalled merlin-systems-0.6.0+20.g521f44a

Successfully installed merlin-systems-0.6.0+20.g521f44a


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 86 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ....                            [ 11%]

tests/unit/systems/dag/test_graph.py ..                                  [ 13%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 16%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 20%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 23%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 24%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 32%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 33%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 69%]

tests/unit/systems/ops/implicit/test_op.py .....F...                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_______________ test_ensemble[AlternatingLeastSquares-ensemble] ________________


model_cls = <function AlternatingLeastSquares at 0x7fb7fd120af0>

backend = 'ensemble'

tmpdir = local('/tmp/pytest-of-jenkins/pytest-18/test_ensemble_AlternatingLeast0')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("backend", ["ensemble", "executor"])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, backend, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema, name=f"{backend}_model")
    triton_ens.export(tmpdir, backend=backend)

    model_name = triton_ens.name
    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None

    with run_triton_server(tmpdir) as client:





      response = client.infer(model_name, inputs, outputs=outputs)





tests/unit/systems/ops/implicit/test_op.py:122:




/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:1322: in infer

raise_error_grpc(rpc_error)




rpc_error = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "in ensemble 'ensemble_model'...of-jenkins/pytest-18/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute\n","grpc_status":13}"






def raise_error_grpc(rpc_error):





  raise get_error_grpc(rpc_error) from None





E       tritonclient.utils.InferenceServerException: [StatusCode.INTERNAL] in ensemble 'ensemble_model', Failed to process the request(s) for model instance '0_predictimplicit', message: TypeError: init(): incompatible constructor arguments. The following argument types are supported:

E           1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)

E

E       Invoked with: kwargs: tensors=[], error="<class 'RuntimeError'>, cublas error: CUBLAS_STATUS_NOT_INITIALIZED (/project/implicit/gpu/knn.cu:87), [<FrameSummary file /tmp/pytest-of-jenkins/pytest-18/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py, line 105 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py, line 59 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py, line 176 in transform>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 87 in recommend>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 122 in knn>, <FrameSummary file _cuda.pyx, line 47 in implicit.gpu._cuda.KnnQuery.cinit>]"

E

E       At:

E         /tmp/pytest-of-jenkins/pytest-18/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute


/usr/local/lib/python3.8/dist-packages/tritonclient/grpc/init.py:62: InferenceServerException

----------------------------- Captured stdout call -----------------------------

Signal (2) received.

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2532.08it/s]

I1021 02:48:31.885915 9227 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f9156000000' with size 268435456

I1021 02:48:31.886983 9227 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864

I1021 02:48:31.890771 9227 model_lifecycle.cc:459] loading: 0_predictimplicit:1

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

I1021 02:48:36.269700 9227 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: 0_predictimplicit (GPU device 0)

I1021 02:48:40.081152 9227 model_lifecycle.cc:693] successfully loaded '0_predictimplicit' version 1

I1021 02:48:40.081472 9227 model_lifecycle.cc:459] loading: ensemble_model:1

I1021 02:48:40.081796 9227 model_lifecycle.cc:693] successfully loaded 'ensemble_model' version 1

I1021 02:48:40.081902 9227 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1021 02:48:40.081990 9227 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1021 02:48:40.082055 9227 server.cc:631]

+-------------------+---------+--------+

| Model             | Version | Status |

+-------------------+---------+--------+

| 0_predictimplicit | 1       | READY  |

| ensemble_model    | 1       | READY  |

+-------------------+---------+--------+


I1021 02:48:40.145517 9227 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1021 02:48:40.146368 9227 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-18/test_ensemble_AlternatingLeast0                                                                                                                             |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 67108864                                                                                                                                                                                     |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1021 02:48:40.147909 9227 grpc_server.cc:4610] Started GRPCInferenceService at localhost:8001

I1021 02:48:40.148124 9227 http_server.cc:3316] Started HTTPService at 0.0.0.0:8000

I1021 02:48:40.189037 9227 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002

W1021 02:48:41.181592 9227 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

1021 02:48:41.195368 9306 pb_stub.cc:777] Failed to process the request(s) for model '0_predictimplicit', message: TypeError: init(): incompatible constructor arguments. The following argument types are supported:

1. c_python_backend_utils.InferenceResponse(output_tensors: List[c_python_backend_utils.Tensor], error: c_python_backend_utils.TritonError = None)


Invoked with: kwargs: tensors=[], error="<class 'RuntimeError'>, cublas error: CUBLAS_STATUS_NOT_INITIALIZED (/project/implicit/gpu/knn.cu:87), [<FrameSummary file /tmp/pytest-of-jenkins/pytest-18/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py, line 105 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/op_runner.py, line 59 in execute>, <FrameSummary file /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ops/implicit.py, line 176 in transform>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 87 in recommend>, <FrameSummary file /usr/local/lib/python3.8/dist-packages/implicit/gpu/matrix_factorization_base.py, line 122 in knn>, <FrameSummary file _cuda.pyx, line 47 in implicit.gpu._cuda.KnnQuery.cinit>]"


At:

/tmp/pytest-of-jenkins/pytest-18/test_ensemble_AlternatingLeast0/0_predictimplicit/1/model.py(122): execute


I1021 02:48:41.196713 9227 server.cc:262] Waiting for in-flight requests to complete.

I1021 02:48:41.196731 9227 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1021 02:48:41.196872 9227 server.cc:293] All models are stopped, unloading models

I1021 02:48:41.196885 9227 server.cc:300] Timeout 30: Found 2 live models and 0 in-flight non-inference requests

I1021 02:48:41.196964 9227 model_lifecycle.cc:578] successfully unloaded 'ensemble_model' version 1

W1021 02:48:42.181809 9227 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

I1021 02:48:42.196969 9227 server.cc:300] Timeout 29: Found 1 live models and 0 in-flight non-inference requests

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(

I1021 02:48:42.693399 9227 model_lifecycle.cc:578] successfully unloaded '0_predictimplicit' version 1

I1021 02:48:43.197102 9227 server.cc:300] Timeout 28: Found 0 live models and 0 in-flight non-inference requests

W1021 02:48:43.207322 9227 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              72     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     14    81%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2288    887    61%


============ 1 failed, 85 passed, 56 warnings in 577.11s (0:09:37) =============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins8279589867490276816.sh

@jperez999
Copy link
Collaborator

jperez999 commented Oct 21, 2022

rerun tests

@nvidia-merlin-bot
Copy link

nvidia-merlin-bot commented Oct 21, 2022

Click to view CI Results
GitHub pull request #204 of commit 521f44a107abd162339805d06faff8fa3620efdd, no merge conflicts.
Running as SYSTEM
Setting status of 521f44a107abd162339805d06faff8fa3620efdd to PENDING with url https://10.20.13.93:8080/job/merlin_systems/450/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/204/*:refs/remotes/origin/pr/204/* # timeout=10
 > git rev-parse 521f44a107abd162339805d06faff8fa3620efdd^{commit} # timeout=10
Checking out Revision 521f44a107abd162339805d06faff8fa3620efdd (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
Commit message: "Merge branch 'main' into refactor/dictarray"
 > git rev-list --no-walk 521f44a107abd162339805d06faff8fa3620efdd # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins9057475041305838497.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/1/merlin-systems-0.6.0+20.g521f44a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,awscli==1.25.95,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.27.94,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cloudpickle==2.2.0,cmake==3.24.1.1,colorama==0.4.4,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.7.1,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.6.0+20.g521f44a,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,pluggy==1.0.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-forked==1.4.0,pytest-xdist==2.5.0,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.9.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.36,stack-data==0.5.0,starlette==0.20.4,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.6.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1800351325'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-_ry2r7es
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-_ry2r7es
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit c405f4719782ee057b29ef2b069731fffb096a50
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (4.64.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (0.55.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (1.10.0)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.5.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.7.0+11.gc405f47) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.7.0+11.gc405f47) (7.0.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.4.1)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.4.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.7.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (3.1.2)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (65.3.0)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.7.0+11.gc405f47) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.7.0+11.gc405f47) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (2022.2.1)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.7.0+11.gc405f47) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.7.0+11.gc405f47) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.7.0+11.gc405f47) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.7.0+11.gc405f47-py3-none-any.whl size=118282 sha256=772a6a8384a76bb1d2c19d1e499b5d86f75f9a68fce2dcda80ceecbc7cc16e55
  Stored in directory: /tmp/pip-ephem-wheel-cache-7tmeegsb/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.7.0+11.gc405f47

[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[1] | python -m pip install .

Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com

Processing /var/jenkins_home/workspace/merlin_systems/systems

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (0.7.0+11.gc405f47)

Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.6.0+20.g521f44a) (2.22.0)

Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.6.0+20.g521f44a) (1.1.1)

Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.19.5)

Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (21.3)

Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.3.5)

Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.64.1)

Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.5)

Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.55.1)

Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.10.0)

Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.5.0)

Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.3.0)

Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (7.0.0)

Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.4.3)

Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.12.0)

Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.2.0)

Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.4.1)

Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (5.8.0)

Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.4.0)

Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.4)

Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.7.0)

Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (8.1.3)

Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.1)

Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.0.0)

Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.1.2)

Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (65.3.0)

Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.20.3)

Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.38.1)

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (3.0.9)

Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.8.2)

Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2022.2.1)

Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.52.0)

Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.2.0)

Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (0.2.1)

Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.15.0)

Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (1.0.1)

Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.1.0)

Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.2)

Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (2.1.1)

Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (6.0.1)

Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.6.0+20.g521f44a) (4.0.0)

Building wheels for collected packages: merlin-systems

Building wheel for merlin-systems (pyproject.toml): started

Building wheel for merlin-systems (pyproject.toml): finished with status 'done'

Created wheel for merlin-systems: filename=merlin_systems-0.6.0+20.g521f44a-py3-none-any.whl size=83592 sha256=3de59c4ba46d4e8cb667977d5afed26d14ee159d8cca7446daaa4e575095954e

Stored in directory: /tmp/pip-ephem-wheel-cache-el779333/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811

Successfully built merlin-systems

Installing collected packages: merlin-systems

Attempting uninstall: merlin-systems

Found existing installation: merlin-systems 0.6.0+20.g521f44a

Uninstalling merlin-systems-0.6.0+20.g521f44a:

Successfully uninstalled merlin-systems-0.6.0+20.g521f44a

Successfully installed merlin-systems-0.6.0+20.g521f44a


[notice] A new release of pip available: 22.2.2 -> 22.3

[notice] To update, run: pip install --upgrade pip

test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit

============================= test session starts ==============================

platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0

cachedir: .tox/test-gpu/.pytest_cache

rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini

plugins: anyio-3.5.0, xdist-2.5.0, forked-1.4.0, cov-4.0.0

collected 86 items


tests/unit/test_export.py .                                              [  1%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  2%]

.                                                                        [  3%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  5%]

[  5%]

tests/unit/systems/dag/test_ensemble.py .                                [  6%]

tests/unit/systems/dag/test_executors.py ....                            [ 11%]

tests/unit/systems/dag/test_graph.py ..                                  [ 13%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 16%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 20%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 23%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 24%]

tests/unit/systems/ops/feast/test_op.py .......                          [ 32%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 33%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 38%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 68%]

tests/unit/systems/ops/implicit/test_executor.py .                       [ 69%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 80%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 81%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 83%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 88%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 93%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[1]

tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view[2]

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                         Stmts   Miss  Cover


merlin/systems/init.py                       6      0   100%

merlin/systems/_version.py                     354    205    42%

merlin/systems/dag/init.py                   3      0   100%

merlin/systems/dag/ensemble.py                 151      8    95%

merlin/systems/dag/node.py                      23      1    96%

merlin/systems/dag/op_runner.py                 26      1    96%

merlin/systems/dag/ops/init.py               9      0   100%

merlin/systems/dag/ops/compat.py                24      8    67%

merlin/systems/dag/ops/faiss.py                 74     20    73%

merlin/systems/dag/ops/feast.py                126      7    94%

merlin/systems/dag/ops/fil.py                  223     34    85%

merlin/systems/dag/ops/implicit.py              72     13    82%

merlin/systems/dag/ops/operator.py              76      5    93%

merlin/systems/dag/ops/pytorch.py               89     12    87%

merlin/systems/dag/ops/session_filter.py        45      7    84%

merlin/systems/dag/ops/softmax_sampling.py      51     17    67%

merlin/systems/dag/ops/tensorflow.py           112     16    86%

merlin/systems/dag/ops/unroll_features.py       50     50     0%

merlin/systems/dag/ops/workflow.py              72     26    64%

merlin/systems/model_registry.py                16      1    94%

merlin/systems/triton/init.py               47     17    64%

merlin/systems/triton/conversions.py            87     73    16%

merlin/systems/triton/export.py                266    155    42%

merlin/systems/triton/utils.py                  72     14    81%

merlin/systems/workflow/init.py             22      5    77%

merlin/systems/workflow/base.py                113    113     0%

merlin/systems/workflow/hugectr.py              37     37     0%

merlin/systems/workflow/pytorch.py              10     10     0%

merlin/systems/workflow/tensorflow.py           32     32     0%


TOTAL                                         2288    887    61%


================= 86 passed, 56 warnings in 589.84s (0:09:49) ==================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins6573902286607233887.sh

@jperez999 jperez999 merged commit 1fb8cab into NVIDIA-Merlin:main Oct 21, 2022
@nv-alaiacano nv-alaiacano added the breaking Breaking change label Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jperez999 jperez999 Awaiting requested review from jperez999

@nv-alaiacano nv-alaiacano Awaiting requested review from nv-alaiacano

1 more reviewer

@oliverholworthy oliverholworthy oliverholworthy approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@karlhigley karlhigley

Labels

breaking Breaking change chore Maintenance for the repository clean up

Projects

None yet

Milestone

Merlin 22.11

Development

Successfully merging this pull request may close these issues.

6 participants

@karlhigley @nvidia-merlin-bot @oliverholworthy @jperez999 @viswa-nvidia @nv-alaiacano