Allow merlin.models.xgb.XGBoost in PredictForest #146
Conversation
The merlin-models XGB wrapper wasn't supported in PredictForest. This change adds basic support for passing the MM xgb wrapper directly to PredictForest (without requiring a dependency on MM itself).
|
@radekosmulski @bschifferer Doing something like this is what I was suggested in the docs sync earlier today |
Click to view CI ResultsGitHub pull request #146 of commit 9a0b7de5153022d79e1461ce71c791fbb3d33ea6, no merge conflicts.
Running as SYSTEM
Setting status of 9a0b7de5153022d79e1461ce71c791fbb3d33ea6 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/147/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
> git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/146/*:refs/remotes/origin/pr/146/* # timeout=10
> git rev-parse 9a0b7de5153022d79e1461ce71c791fbb3d33ea6^{commit} # timeout=10
Checking out Revision 9a0b7de5153022d79e1461ce71c791fbb3d33ea6 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 9a0b7de5153022d79e1461ce71c791fbb3d33ea6 # timeout=10
Commit message: "Allow merlin.models.xgb.XGBoost in PredictForest"
> git rev-list --no-walk 92070d02437d7679280097b7eaf495c1f5b19541 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins6548990256094765950.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 49 items
|
Click to view CI ResultsGitHub pull request #146 of commit cc2aea6aa1fda4ef8be402aabcfcbef449c0fd32, no merge conflicts.
Running as SYSTEM
Setting status of cc2aea6aa1fda4ef8be402aabcfcbef449c0fd32 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/148/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
> git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/146/*:refs/remotes/origin/pr/146/* # timeout=10
> git rev-parse cc2aea6aa1fda4ef8be402aabcfcbef449c0fd32^{commit} # timeout=10
Checking out Revision cc2aea6aa1fda4ef8be402aabcfcbef449c0fd32 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f cc2aea6aa1fda4ef8be402aabcfcbef449c0fd32 # timeout=10
Commit message: "flake8"
> git rev-list --no-walk 9a0b7de5153022d79e1461ce71c791fbb3d33ea6 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins6513512587687028572.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 49 items
|
|
@benfred this looks great! |
|
CI is failing here - but will be fixed by #147 |
Documentation preview |
Click to view CI ResultsGitHub pull request #146 of commit 9910bfe09817c83159a425874220a69c59878d57, no merge conflicts.
Running as SYSTEM
Setting status of 9910bfe09817c83159a425874220a69c59878d57 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/151/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
> git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/146/*:refs/remotes/origin/pr/146/* # timeout=10
> git rev-parse 9910bfe09817c83159a425874220a69c59878d57^{commit} # timeout=10
Checking out Revision 9910bfe09817c83159a425874220a69c59878d57 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 9910bfe09817c83159a425874220a69c59878d57 # timeout=10
Commit message: "Merge branch 'main' into merlin_models_xgboost"
> git rev-list --no-walk 0c51ffcf3d4a1d0d4f8eadf54a966ce6db8991b5 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins8394693905007362483.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 49 items
|
Click to view CI ResultsGitHub pull request #146 of commit 6ebf6cba52aa6dc19397f0ff9c451d63a86b5dd7, no merge conflicts.
Running as SYSTEM
Setting status of 6ebf6cba52aa6dc19397f0ff9c451d63a86b5dd7 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/166/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
> git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/146/*:refs/remotes/origin/pr/146/* # timeout=10
> git rev-parse 6ebf6cba52aa6dc19397f0ff9c451d63a86b5dd7^{commit} # timeout=10
Checking out Revision 6ebf6cba52aa6dc19397f0ff9c451d63a86b5dd7 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6ebf6cba52aa6dc19397f0ff9c451d63a86b5dd7 # timeout=10
Commit message: "Merge branch 'main' into merlin_models_xgboost"
> git rev-list --no-walk a1c662b2c7770cbeba84910f9c75c5abce939a70 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins16374347690534643245.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 52 items
|
Click to view CI ResultsGitHub pull request #146 of commit 6c2abeca8c03886b081b4acbbf3d41bb2a697aa3, no merge conflicts.
Running as SYSTEM
Setting status of 6c2abeca8c03886b081b4acbbf3d41bb2a697aa3 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/181/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
> git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/146/*:refs/remotes/origin/pr/146/* # timeout=10
> git rev-parse 6c2abeca8c03886b081b4acbbf3d41bb2a697aa3^{commit} # timeout=10
Checking out Revision 6c2abeca8c03886b081b4acbbf3d41bb2a697aa3 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 6c2abeca8c03886b081b4acbbf3d41bb2a697aa3 # timeout=10
Commit message: "Merge branch 'main' into merlin_models_xgboost"
> git rev-list --no-walk c527395faea83392dcf219f2064150331aaf482d # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins10909051296101069266.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 53 items
|
| ) | ||
|
|
||
| with open(node_export_path / "config.pbtxt", "w") as o: | ||
| with open(node_export_path / "config.pbtxt", "w", encoding="utf-8") as o: |
There was a problem hiding this comment.
pre-commit now checks encoding specified in open (probably since #150)
Click to view CI ResultsGitHub pull request #146 of commit 8898b039c830cf0f826fd044164311bf72190883, no merge conflicts.
Running as SYSTEM
Setting status of 8898b039c830cf0f826fd044164311bf72190883 to PENDING with url https://10.20.13.93:8080/job/merlin_systems/183/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
> git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/146/*:refs/remotes/origin/pr/146/* # timeout=10
> git rev-parse 8898b039c830cf0f826fd044164311bf72190883^{commit} # timeout=10
Checking out Revision 8898b039c830cf0f826fd044164311bf72190883 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 8898b039c830cf0f826fd044164311bf72190883 # timeout=10
Commit message: "Move check for merlin.models.xgb.XGBoost up to other XGBoost checks"
> git rev-list --no-walk 091b6da9c68908344e9a361a4f22574ea571bdd2 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins2849312034288256604.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 53 items
|
oliverholworthy
dismissed
EvenOldridge’s stale review
requested changes addressed in 8898b03
| assert parsed_config.backend == "fil" | ||
|
|
||
|
|
||
| def test_export_merlin_models(tmpdir): |
There was a problem hiding this comment.
This test only runs in Jenkins for now, can be activated in CPU / Github actions workflow by adding merlin models install in setup
The merlin-models XGB wrapper wasn't supported in PredictForest. This change
adds basic support for passing the MM xgb wrapper directly to PredictForest
(without requiring a dependency on MM itself).