From d390fe62d14c16d182595e70add41d0b8b0400e5 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 13 Aug 2025 11:45:09 -0500 Subject: [PATCH 1/3] post-processing: update and expand docs --- ci/README.md | 66 +++++++++++++++ ci/customization/README.md | 159 ++++++++++++++++++++++++------------- 2 files changed, 168 insertions(+), 57 deletions(-) create mode 100644 ci/README.md diff --git a/ci/README.md b/ci/README.md new file mode 100644 index 00000000000..6ca14c59b58 --- /dev/null +++ b/ci/README.md @@ -0,0 +1,66 @@ +# ci + +## API docs + +The site's `/api` page links to API documentation for many RAPIDS projects. + +Those API docs are built in CI of the individual projects, published to object storage, +then pulled into builds here and integrated into the rest of the docs site. + +The steps are roughly as follows. + +### Step 1: determine libraries and versions to build + +`get-projects-to-versions.sh` is responsible for holding all the logic relevant to the question "projects' API docs should be hosted? what versions?". + +Run it from the root of the repo to see for yourself. + +```shell +./ci/get-projects-to-versions.sh +``` + +That script is re-used by other automation to determine which projects and versions to build. + +### Step 2: download API docs + +All of the API docs for RAPIDS projects are uploaded to an S3 bucket from those projects' CI. + +To integrate them into the docs site, they need to be downloaded and placed in `_site/api/` locally. + +```shell +./ci/download_from_s3.sh +``` + +### Step 3: post-processing + +After downloading the API docs, they'll all be arranged locally in directories named with version numbers, like this: + +```text +_site/api/ +├── cudf +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +├── cudf-java +│ ├── 25.06 +│ ├── 25.08 +├── cuml +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +``` + +Further processing is then needed to integrate them into the docs site, including: + +* mapping version numbers like `25.10` to version identifies like `"stable"` (to populate links like https://docs.rapids.ai/api/cudf/stable/) +* adding version selectors on all the docs pages +* other formatting and visual consistency changes + +That is handled by the code in the `ci/customization` directory. +You can invoke it all like this: + +```shell +./ci/post-process.sh +``` + +See the docs in `ci/customization` for more details ([customization/README.md](./customization/README.md)). diff --git a/ci/customization/README.md b/ci/customization/README.md index 9e4b44d4b8b..5c858dece07 100644 --- a/ci/customization/README.md +++ b/ci/customization/README.md @@ -1,10 +1,15 @@ # Documentation Customization -The documentation below describes how the scripts work to customize the generated HTML files. The scripts should be run in the order they appear in the table of contents. +The documentation below describes how the scripts work to customize the generated HTML files. +The scripts should be run in the order they appear in the table of contents. + +Unless otherwise stated, all scripts should be run from the root of this repository. ## Table of Contents - [Dependencies](#dependencies) +- [Quickstart](#quckstart) +- [Project Versions](#project-versions) - [Update Symlinks](#symlinks) - [Generate Library Map](#generate-library-map) - [Customize HTML File](#customize-html) @@ -12,35 +17,94 @@ The documentation below describes how the scripts work to customize the generate ### Dependencies -- [jq](https://github.com/stedolan/jq) -- [BeautifulSoup4](https://github.com/waylan/beautifulsoup) +- `aws` CLI ([docs](https://aws.amazon.com/cli/)) +- `jq` ([docs](https://jqlang.org/manual/)) +- Python / `pip` +- `yq` ([docs](https://mikefarah.gitbook.io/yq)) + +### Quickstart + +If you just want to try running all of the customization locally, download all of the API docs ([as described in `ci/README.md`](../README.md)), then invoke the following script: + +```shell +./ci/post-process.sh +``` + +To learn more about each step, keep reading. + +### Project Versions + +Building the API docs requires answering these questions: + +* which RAPIDS projects? +* what version types? (stable? legacy? nightly?) +* what version numbers correspond to those version types? + +Logic for all of that is centralized in a script. + +Invoke it to see what will be built. + +```shell +./ci/get-projects-to-versions.sh > ./ci/customization/projects-to-versions.json +``` ### Symlinks -For the doc customization to work correctly, all of the symlinks need to be up to date. The symlinks ultimately enable us to generate paths like `/cudf/stable/` that point to the latest RAPIDS release version folder (i.e. `/cudf/0.13/`). +Initially, the downloaded API docs are in directories named with version numbers, like this: -[update_symlinks.sh](/update_symlinks.sh) is a shell script that updates each project's symlinks according to the versions in [releases.json](/_data/releases.json). It looks for the legacy, stable, and nightly version folders (i.e. `0.11`, `0.12`, etc.) and creates the corresponding symlinks if those folders exist. +```console +$ tree -L 2 ./_site/api +_site/api/ +├── cudf +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +``` -**Usage:** +Those directory paths become links on the docs site, like https://docs.rapids.ai/api/cudf/25.10/. -> **Note:** This script is intended to be run from the project's root. +To be more bookmark-friendly, the docs site also has links with version _types_ like `/legacy`, `/nightly`, and `/stable`. -```sh -update_symlinks.sh +To avoid fully copying files, that's done via creating symlinks, similar to this: + +```shell +pushd ./_site/api/cudf +ln -s 25.10 stable ``` -### Generate Library Map +To avoid hard-coding all those versions and paths, that logic is automated in a script. +Try running it to generate the symlinks. + +```shell +./ci/update_symlinks.sh ./ci/customization/projects-to-versions.json +``` + +After that is run, you'll see links like the following have been created. + +```console +$ tree -L 2 ./_site/api +./_site/api/ +├── cudf +│ ├── 25.06 +│ ├── 25.08 +│ ├── 25.10 +│ ├── latest -> 25.08 +│ ├── legacy -> 25.06 +│ ├── nightly -> 25.10 +│ └── stable -> 25.08 +``` -One of the ways we customize the documentation files is by adding dropdown selectors to each doc page that allows visitors to select and navigate between RAPIDS libraries and their versions. In order to know what page a user will be directed to upon selection, a map of available libraries and versions needs to be generated. [lib_map.sh](lib_map.sh) generates this map. +### Generate Library Map -`lib_map.sh` creates a JSON file that is used by `customize_docs.py` to populate the available options for the _libray_ and _version_ selectors. +One of the ways we customize the documentation files is by adding dropdown selectors to each doc page that allows visitors to select and navigate between RAPIDS libraries and their versions. -**Usage:** +In order to know what page a user will be directed to upon selection, a map of available libraries and versions needs to be generated. +[lib_map.sh](lib_map.sh) generates this map. -> **Note:** This script is intended to be run from the project's root. +`lib_map.sh` creates a JSON file that is used by `customize_docs.py` to populate the available options for the _library_ and _version_ selectors. ```sh -customization/lib_map.sh +./ci/customization/lib_map.sh ``` An excerpt of the generated JSON file is shown below: @@ -48,26 +112,28 @@ An excerpt of the generated JSON file is shown below: ```json { "cuml": { - "nightly": "/cuml/en/nightly/api.html", - "stable": "/cuml/en/stable/api.html", - "legacy": "/cuml/en/legacy/api.html" + "nightly": "/cuml/en/nightly", + "stable": "/cuml/en/stable", + "legacy": "/cuml/en/legacy" + }, + "libcudf": { + "nightly": "/libcudf/nightly/namespacecudf/", + "stable": "/libcudf/stable/namespacecudf/", + "legacy": "/libcudf/legacy/namespacecudf/" }, - "cuxfilter": { - "nightly": "/cuxfilter/en/nightly", + "rapidsmpf": { + "nightly": "/api/rapidsmpf/nightly", "stable": null, "legacy": null }, - "libcudf": { - "nightly": "/libcudf/nightly/namespacecudf.html", - "stable": "/libcudf/stable/namespacecudf.html", - "legacy": "/libcudf/legacy/namespacecudf.html" - } } ``` ### Customize HTML -[customize_doc.py](customize_doc.py) is a Python script that uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) to theme Doxygen and Sphinx HTML documentation files. It adds the following elements to each documentation page: +[customize_doc.py](customize_doc.py) is a Python script that uses [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) to theme Doxygen and Sphinx HTML documentation files. + +It adds the following elements to each documentation page: - A _Home_ button that links to https://docs.rapids.ai/api - A _library_ selector that enables navigation to different RAPIDS libraries @@ -78,39 +144,18 @@ An excerpt of the generated JSON file is shown below: `customize_doc.py` can be safely run multiple times on a single file without adding duplicate elements to the page. -**Usage:** - -> Before running the script, it is important that [update_symlinks.sh](/update_symlinks.sh) and [lib_map.sh](lib_map.sh) have been run. +That's invoked by a wrapper script that determines HTML files to pass in. ```sh -python customization/customize_docs.py ${ABS_PATH_TO_HTML_FILE} ${CURRENT_RAPIDS_VERSION} -``` - -#### Helper Script - -[customize_docs_in_folder.sh](customize_docs_in_folder.sh) is a helper script that makes it easy to recursively customize all of the files in a specified folder. - -**Usage:** - -> **Note:** This script is intended to be run from the project's root. - -```sh -customization/customize_docs_in_folder.sh api/ - -# or - -customization/customize_docs_in_folder.sh api/rmm -``` - -### TL;DR - -To customize the docs, run: - -```sh -update_symlinks.sh # ensures symlink accuracy - -customization/lib_map.sh # generates a JSON file needed by customize_docs.py +PROJECTS_TO_JSON_PATH="./ci/customization/projects-to-versions.json" -customization/customize_docs_in_folder.sh api/rmm +# all API docs +./ci/customization/customize_docs_in_folder.sh \ + ./_site/api \ + "${PROJECTS_TO_JSON_PATH}" +# just cudf +./ci/customization/customize_docs_in_folder.sh \ + ./site/api/cudf \ + ./ci/customization/projects-to-versions.json ``` From 3344cb36261f48b5d9da5ed3851e2120205f6c73 Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 13 Aug 2025 11:49:46 -0500 Subject: [PATCH 2/3] typo --- ci/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/README.md b/ci/README.md index 6ca14c59b58..da05b77d00d 100644 --- a/ci/README.md +++ b/ci/README.md @@ -19,7 +19,7 @@ Run it from the root of the repo to see for yourself. ./ci/get-projects-to-versions.sh ``` -That script is re-used by other automation to determine which projects and versions to build. +That script is reused by other automation to determine which projects and versions to build. ### Step 2: download API docs From 354785f90efe1d17858a198cca7a5a0d95eedf4b Mon Sep 17 00:00:00 2001 From: James Lamb Date: Wed, 13 Aug 2025 13:14:55 -0500 Subject: [PATCH 3/3] Update ci/README.md Co-authored-by: Bradley Dice --- ci/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci/README.md b/ci/README.md index da05b77d00d..28809a58a18 100644 --- a/ci/README.md +++ b/ci/README.md @@ -11,7 +11,7 @@ The steps are roughly as follows. ### Step 1: determine libraries and versions to build -`get-projects-to-versions.sh` is responsible for holding all the logic relevant to the question "projects' API docs should be hosted? what versions?". +`get-projects-to-versions.sh` is responsible for holding all the logic relevant to the questions "What projects' API docs should be hosted? What versions?". Run it from the root of the repo to see for yourself.