add libcugraph Python builds by jameslamb · Pull Request #435 · rapidsai/devcontainers

jameslamb · 2025-01-10T16:47:54Z

Contributes to rapidsai/build-planning#33

Adjusts rapids-build-utils manifest for release 25.02 to account for the introduction of new libcugraph wheels (rapidsai/cugraph#4804).

Notes for Reviewers

This shouldn't be merged still pointing at my forks. Plan:

admin-merge introduce libcugraph wheels cugraph#4804 once everything except devcontainers CI there is passing
point this PR at upstream rapidsai/cugraph
observe CI passing and merge this normally (or admin-merge to save time)

jameslamb · 2025-01-14T17:40:38Z

I think CI here will be broken until rapidsai/cudf#17733 is merged.

Because of rapidsai/kvikio#581

Replaces #2306, contributes to rapidsai/build-planning#33. Proposes packaging `libraft` as a wheel, which is then re-used by: * `pylibraft-cu{11,12}` and `raft-cu{11,12}` (this PR) * `libcugraph-cu{11,12}`, `pylibcugraph-cu{11,12}`, and `cugraph-cu{11,12}` in rapidsai/cugraph#4804 * `libcuml-cu{11,12}` and `cuml-cu{11,12}` in rapidsai/cuml#6199 As part of this, also proposes: * introducing a new CMake option, `RAFT_COMPILE_DYNAMIC_ONLY`, to allow building/installing only the dynamic shared library (i.e. skipping the static library) * enforcing `rapids-cmake`'s preferred CMake style (#2531 (comment)) * making wheel-building CI jobs always depend on other wheel-building CI jobs, not tests or `*-publish` (to reduce end-to-end CI time) ## Notes for Reviewers ### Benefits of these changes * smaller wheels (see "Size Changes" below) * faster compile times (no more re-compiling RAFT in cuGraph and cuML CI) * other benefits mentioned in rapidsai/build-planning#33 ### Wheel contents `libraft`: * `libraft.so` (shared library) * RAFT headers * vendored dependencies (`fmt`, CCCL, `cuco`, `cute`, `cutlass`) `pylibraft`: * `pylibraft` Python / Cython code and compiled Cython extensions `raft-dask`: * `raft-dask` Python / Cython code and compiled Cython extension ### Dependency Flows In short.... `libraft` contains a `libraft.so` dynamic library and the headers to link against it. * Anything that needs to link against RAFT at build time pulls in `libraft` wheels as a build dependency. * Anything that needs RAFT's symbols at runtime pulls it in as a runtime dependency, and calls `libraft.load_library()`. For more details and some flowcharts, see rapidsai/build-planning#33 (comment) ### Size changes (CUDA 12, Python 3.12, x86_64) | wheel | num files (before) | num files (these PRs) | size (before) | size (these PRs) | |:---------------:|------------------:|-----------------:|--------------:|-------------:| | `libraft`. | --- | 3169 | --- | 19M | | `pylibraft` | 64 | 63 | 11M | 1M | | `raft-dask` | 29 | 28 | 188M | 188M | | `libcugraph` | --- | 1762 | --- | 903M | | `pylibcugraph` | 190 | 187 | 901M | 2M | | `cugraph` | 315 | 313 | 899M | 3.0M | | `libcuml` | --- | 1766 | --- | 289M | | `cuml` | 442 | --- | 517M | --- | |**TOTAL** | **1,040** | **7,268** | **2,516M** | **1,405M** | *NOTES: size = compressed, "before" = 2025-01-13 nightlies* <details><summary>how I calculated those (click me)</summary> * `cugraph`: nightly commit = rapidsai/cugraph@8507cbf, PR = rapidsai/cugraph#4804 * `cuml`: nightly commit = rapidsai/cuml@7c715c4, PR = rapidsai/cuml#6199 * `raft`: nightly commit = 1b62c41, PR = this PR ```shell docker run \ --rm \ --network host \ --env RAPIDS_NIGHTLY_DATE=2025-01-13 \ --env CUGRAPH_NIGHTLY_SHA=8507cbf63db2f349136b266d3e6e787b189f45a0 \ --env CUGRAPH_PR="pull-request/4804" \ --env CUGRAPH_PR_SHA="2ef32eaa006a84c0bd16220bb8e8af34198fbee8" \ --env CUML_NIGHTLY_SHA=7c715c494dff71274d0fdec774bdee12a7e78827 \ --env CUML_PR="pull-request/6199" \ --env CUML_PR_SHA="2ef32eaa006a84c0bd16220bb8e8af34198fbee8" \ --env RAFT_NIGHTLY_SHA=1b62c4117a35b11ce3c830daae248e32ebf75e3f \ --env RAFT_PR="pull-request/2531" \ --env RAFT_PR_SHA="0d6597b08919f2aae8ac268f1a68d6a8fe5beb4e" \ --env RAPIDS_PY_CUDA_SUFFIX=cu12 \ --env WHEEL_DIR_BEFORE=/tmp/wheels-before \ --env WHEEL_DIR_AFTER=/tmp/wheels-after \ -it rapidsai/ci-wheel:cuda12.5.1-rockylinux8-py3.12 \ bash # --- nightly wheels --- # mkdir -p ./wheels-before export RAPIDS_BUILD_TYPE=branch export RAPIDS_REF_NAME="branch-25.02" # pylibraft RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_SHA=${RAFT_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # raft-dask RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_SHA=${RAFT_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # cugraph RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_SHA=${CUGRAPH_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # pylibcugraph RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_SHA=${CUGRAPH_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # cuml RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuml \ RAPIDS_SHA=${CUML_NIGHTLY_SHA} \ rapids-download-wheels-from-s3 python ./wheels-before # --- wheels from CI --- # mkdir -p ./wheels-after export RAPIDS_BUILD_TYPE="pull-request" # libraft RAPIDS_PY_WHEEL_NAME="libraft_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_REF_NAME="${RAFT_PR}" \ RAPIDS_SHA="${RAFT_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # pylibraft RAPIDS_PY_WHEEL_NAME="pylibraft_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_REF_NAME="${RAFT_PR}" \ RAPIDS_SHA="${RAFT_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # raft-dask RAPIDS_PY_WHEEL_NAME="raft_dask_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/raft \ RAPIDS_REF_NAME="${RAFT_PR}" \ RAPIDS_SHA="${RAFT_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # libcugraph RAPIDS_PY_WHEEL_NAME="libcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_REF_NAME="${CUGRAPH_PR}" \ RAPIDS_SHA="${CUGRAPH_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # pylibcugraph RAPIDS_PY_WHEEL_NAME="pylibcugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_REF_NAME="${CUGRAPH_PR}" \ RAPIDS_SHA="${CUGRAPH_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # cugraph RAPIDS_PY_WHEEL_NAME="cugraph_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cugraph \ RAPIDS_REF_NAME="${CUGRAPH_PR}" \ RAPIDS_SHA="${CUGRAPH_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after # libcuml RAPIDS_PY_WHEEL_NAME="libcuml_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuml \ RAPIDS_REF_NAME="${CUML_PR}" \ RAPIDS_SHA="${CUML_PR_SHA}" \ rapids-download-wheels-from-s3 cpp ./wheels-after # cuml RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" \ RAPIDS_REPOSITORY=rapidsai/cuml \ RAPIDS_REF_NAME="${CUML_PR}" \ RAPIDS_SHA="${CUML_PR_SHA}" \ rapids-download-wheels-from-s3 python ./wheels-after pip install pydistcheck pydistcheck \ --inspect \ --select 'distro-too-large-compressed' \ ./wheels-before/*.whl \ | grep -E '^checking|files: | compressed' \ > ./before.txt # get more exact sizes du -sh ./wheels-before/* pydistcheck \ --inspect \ --select 'distro-too-large-compressed' \ ./wheels-after/*.whl \ | grep -E '^checking|files: | compressed' \ > ./after.txt # get more exact sizes du -sh ./wheels-after/* ``` </details> ### How I tested this These other PRs: * rapidsai/devcontainers#435 * rapidsai/cugraph-gnn#110 * rapidsai/cuml#6199 * rapidsai/cugraph#4804

bdice · 2025-01-17T14:22:05Z

I reverted the RAFT testing changes now that #438 is in.

trxcllnt

Requesting changes so we don't forget to revert the TODO before merging.

jameslamb · 2025-01-17T19:28:46Z

🎉 build succeeded (build link)

That's a good sign that the changes from rapidsai/cugraph#4804 would be safe to merge (from the devcontainers perspective).

bdice

I think I have power to merge this and unblock cugraph’s PR without requiring an admin merge over the weekend for cugraph. I am going to give that a try. If it doesn’t work out, we can fix it next week.

…t.yaml Co-authored-by: Paul Taylor <178183+trxcllnt@users.noreply.github.com>

Addressed reversion.

Replaces #4340, contributes to rapidsai/build-planning#33. Proposes packaging `libcugraph` as a wheel, which is then re-used by `cugraph-cu{11,12}` and `pylibcugraph-cu{11,12}` wheels. ## Notes for Reviewers ### Benefits of these changes * smaller wheels (see "Size Changes" below) - *no more `pylibcugraph` and `cugraph` both holding copies of libcugraph.so* * faster compile times - *no more re-compiling RAFT, thanks to rapidsai/raft#2531 - *no more recompiling libcugraph.so in both `pylibcugraph` and `cugraph` wheel builds* * other benefits mentioned in rapidsai/build-planning#33 ### Wheel contents `libcugraph`: * `libcugraph.so` (shared library) * cuGraph headers * vendored dependencies (`fmt`, `spdlog`, CCCL, `cuco`) `pylibcugraph`: * `pylibcugraph` Python / Cython code and compiled Cython extensions `cugraph`: * `cugraph` Python / Cython code and compiled Cython extension ### Dependency Flows In short.... `libcugraph` contains `libcugraph.so` and `libcugraph_c.so` dynamic libraries and the headers to link against it. * Anything that needs to link against cuGraph at build time pulls in `libcugraph` wheels as a build dependency. * Anything that needs cuGraph's symbols at runtime pulls it in as a runtime dependency, and calls `libcugraph.load_library()`. For more details and some flowcharts, see rapidsai/build-planning#33 (comment) ### Size changes (CUDA 12, Python 3.12, x86_64) | wheel | num files (before) | num files (this PR) | size (before) | size (this PR) | |:---------------:|------------------:|-----------------:|--------------:|-------------:| | `libcugraph` | --- | 1762 | --- | 903M | | `pylibcugraph` | 190 | 187 | 901M | 2M | | `cugraph` | 315 | 313 | 899M | 3M | |**TOTAL** | **505** | **2,262** | **1,800M** | **908M** | *NOTES: size = compressed, "before" = 2025-01-13 nightlies* *This is a cuGraph-specific slice of the table from rapidsai/raft#2531. See that PR for details.* ### How I tested this These other PRs: * rapidsai/devcontainers#435 * rapidsai/cugraph-gnn#110 Authors: - James Lamb (https://github.com/jameslamb) - Ralph Liu (https://github.com/nv-rliu) - Bradley Dice (https://github.com/bdice) Approvers: - Brad Rees (https://github.com/BradReesWork) - Bradley Dice (https://github.com/bdice) URL: #4804

add libraft, libcugraph, libcuml Python builds

8496eda

jameslamb added breaking Introduces a breaking change improvement Improves an existing functionality DO NOT MERGE labels Jan 10, 2025

This was referenced Jan 10, 2025

introduce libraft wheels rapidsai/raft#2531

Merged

introduce libcugraph wheels rapidsai/cugraph#4804

Merged

empty commit to re-trigger CI

3488563

jameslamb added 2 commits January 14, 2025 15:53

empty commit to re-trigger CI

5cc60c5

empty commit to re-trigger CI

380c8df

bdice added 2 commits January 17, 2025 08:20

Merge remote-tracking branch 'upstream/branch-25.02' into new-cpp-libs

ac5cdd3

Remove RAFT testing changes.

bb3a8ee

bdice changed the title ~~WIP: add libraft, libcugraph, libcuml Python builds~~ WIP: add libcugraph, libcuml Python builds Jan 17, 2025

remove cuML stuff

25a6483

jameslamb changed the title ~~WIP: add libcugraph, libcuml Python builds~~ WIP: add libcugraph Python builds Jan 17, 2025

jameslamb changed the title ~~WIP: add libcugraph Python builds~~ add libcugraph Python builds Jan 17, 2025

jameslamb requested a review from bdice January 17, 2025 16:50

jameslamb marked this pull request as ready for review January 17, 2025 16:50

jameslamb requested a review from a team as a code owner January 17, 2025 16:50

bdice approved these changes Jan 17, 2025

View reviewed changes

trxcllnt previously requested changes Jan 17, 2025

View reviewed changes

Comment thread features/src/rapids-build-utils/opt/rapids-build-utils/manifest.yaml Outdated

bdice approved these changes Jan 18, 2025

View reviewed changes

bdice and others added 2 commits January 17, 2025 19:06

Update features/src/rapids-build-utils/opt/rapids-build-utils/manifes…

12a7ea9

…t.yaml Co-authored-by: Paul Taylor <178183+trxcllnt@users.noreply.github.com>

Merge branch 'branch-25.02' into new-cpp-libs

80f9655

bdice merged commit 5fb9720 into rapidsai:branch-25.02 Jan 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add libcugraph Python builds#435

add libcugraph Python builds#435
bdice merged 9 commits intorapidsai:branch-25.02from
jameslamb:new-cpp-libs

jameslamb commented Jan 10, 2025 •

edited

Loading

Uh oh!

jameslamb commented Jan 14, 2025

Uh oh!

bdice commented Jan 17, 2025

Uh oh!

trxcllnt left a comment

Uh oh!

Uh oh!

jameslamb commented Jan 17, 2025

Uh oh!

bdice left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jameslamb commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes for Reviewers

Uh oh!

jameslamb commented Jan 14, 2025

Uh oh!

bdice commented Jan 17, 2025

Uh oh!

trxcllnt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jameslamb commented Jan 17, 2025

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jameslamb commented Jan 10, 2025 •

edited

Loading