From a66326525b7eff3744d6b73ead98828525f1ab2a Mon Sep 17 00:00:00 2001 From: Aaditi Rai Date: Tue, 17 Mar 2026 18:43:54 -0400 Subject: [PATCH 1/5] Accessibility changes for Containers Workshop --- content/notes/containers-for-hpc/_index.md | 8 +- .../containers-for-hpc/building-apptainer.md | 37 +++++----- .../containers-for-hpc/building-docker.md | 44 +++++------ content/notes/containers-for-hpc/intro.md | 8 +- content/notes/containers-for-hpc/minimal.md | 74 +++++++++++-------- content/notes/containers-for-hpc/using.md | 46 ++++-------- 6 files changed, 97 insertions(+), 120 deletions(-) diff --git a/content/notes/containers-for-hpc/_index.md b/content/notes/containers-for-hpc/_index.md index c838cc65..61015558 100644 --- a/content/notes/containers-for-hpc/_index.md +++ b/content/notes/containers-for-hpc/_index.md @@ -5,18 +5,14 @@ authors: [rs] categories: ["Containers","HPC"] tags: [Containers, HPC] toc: true -type: docs +type: book date: "2025-05-15T00:00:00" weight: 1 -menu: - containers-for-hpc: - name: Software Containers for HPC - --- ## Overview -This tutorial is an introduction to using and building software containers, structured as follows: +This short course is an introduction to using and building software containers, structured as follows: - Introduction to Software Containers - Using Containers on HPC [Apptainer] diff --git a/content/notes/containers-for-hpc/building-apptainer.md b/content/notes/containers-for-hpc/building-apptainer.md index 0670a480..a0b348f6 100644 --- a/content/notes/containers-for-hpc/building-apptainer.md +++ b/content/notes/containers-for-hpc/building-apptainer.md @@ -2,11 +2,8 @@ date: "2025-06-04T00:00:00" title: "Building Containers on HPC [Apptainer]" toc: true -type: docs +type: book weight: 4 -menu: - containers-for-hpc: - parent: Software Containers for HPC --- @@ -114,7 +111,7 @@ This is the very first entry. It defines the bootstrap agent: - `docker` - `library` - `shub` -- and [many more](https://apptainer.org/docs/user/latest/definition_files.html#preferred-bootstrap-agents) +- and [many more (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html#preferred-bootstrap-agents) #### `From` (mandatory) Define the base container. @@ -342,7 +339,7 @@ Save this as `lolcow_2.def`. - You may need to specify extra packages - `fortune` itself provides the executable without the message database - `fortunes-min` contains the message database -- See [how Ubuntu reduced image size by 60%](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) +- See [how Ubuntu reduced image size by 60% (opens in new tab)](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) ### Image size comparison @@ -447,21 +444,21 @@ $ apptainer push lolcow_0.sif oras://myname/lolcow:0 ``` Check: -`https://hub.docker.com/r/myname/lolcow/tags` +[Docker Hub tags for `myname/lolcow` (opens in new tab)](https://hub.docker.com/r/myname/lolcow/tags) -{{< info >}}Best practice: Sign your containers; see [here](https://apptainer.org/docs/user/latest/signNverify.html).{{< /info >}} +{{< info >}}Best practice: Sign your containers; see [Apptainer signing and verification documentation (opens in new tab)](https://apptainer.org/docs/user/latest/signNverify.html).{{< /info >}} {{< warning >}}While Apptainer can convert a Docker image into SIF, you cannot run SIF with Docker. You are simply using Docker Hub to host your SIF - it is not converted into Docker.{{< /warning >}} #### UVA Research Computing container resources -[Docker Hub account](https://hub.docker.com/u/uvarc) +[Docker Hub account (opens in new tab)](https://hub.docker.com/u/uvarc) -[Repository of Dockerfiles and definition files](https://github.com/uvarc/rivanna-docker) +[Repository of Dockerfiles and definition files (opens in new tab)](https://github.com/uvarc/rivanna-docker) ### GitHub Packages -1. [Create a personal access token](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic) +1. [Create a personal access token (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic) 1. Login ```bash apptainer remote login --username myname docker://ghcr.io @@ -471,7 +468,7 @@ Check: ```bash apptainer push lolcow_0.sif oras://ghcr.io/myname/lolcow ``` -1. Check `https://github.com/users/myname/packages/container/package/lolcow` +1. Check [your GitHub package page (opens in new tab)](https://github.com/users/myname/packages/container/package/lolcow) ## Case Studies @@ -487,16 +484,16 @@ While you can create a conda environment locally, you cannot directly migrate it Your project requires PyTorch, Numpy, Seaborn, and Pandas. Write the corresponding Apptainer definition file. Hints: -- Find the appropriate base image from [here](https://hub.docker.com/r/pytorch/pytorch/tags). +- Find the appropriate base image from [PyTorch Docker image tags on Docker Hub (opens in new tab)](https://hub.docker.com/r/pytorch/pytorch/tags). - Pull the base image and examine it first. Does it already provide some packages? {{< info >}}While PyTorch runs on a GPU, you do not need to build the container on a GPU.{{< /info >}} -{{< info >}}You will likely run out of memory when building large containers (over a few GBs). Request an [interactive job](https://www.rc.virginia.edu/userinfo/rivanna/slurm/#submitting-an-interactive-job) with say `--mem=50G` to build on a compute node with 50 GB memory.{{< /info >}} +{{< info >}}You will likely run out of memory when building large containers (over a few GBs). Request an [interactive job (opens in new tab)](https://www.rc.virginia.edu/userinfo/rivanna/slurm/#submitting-an-interactive-job) with say `--mem=50G` to build on a compute node with 50 GB memory.{{< /info >}} ### R -Rocker provides many base images for all R versions (see [here](https://rocker-project.org/images/)): +Rocker provides many base images for all R versions (see [Rocker image catalog (opens in new tab)](https://rocker-project.org/images/)): - `rocker/r-ver`: basic R installation - `rocker/rstudio`: with RStudio Server - `rocker/tidyverse`: plus tidyverse and dependencies @@ -515,14 +512,14 @@ Hints: ## Multistage Build -By distinguishing between buildtime-dependencies vs runtime-dependencies, it is possible to reduce the image size drastically via a [multistage build](https://apptainer.org/docs/user/latest/definition_files.html#multi-stage-builds). (My experience with some extreme cases is that only 1% is needed at runtime.) +By distinguishing between buildtime-dependencies vs runtime-dependencies, it is possible to reduce the image size drastically via a [multistage build (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html#multi-stage-builds). (My experience with some extreme cases is that only 1% is needed at runtime.) This is beyond the scope of the workshop, but you are welcome to browse the Apptainer documentation and Appendix 2 on Minimal Containers. --- ## References -- [Apptainer User Guide](https://apptainer.org/docs/user/latest) -- [Definition File](https://apptainer.org/docs/user/latest/definition_files.html) -- [Definition File vs Dockerfile](https://apptainer.org/docs/user/latest/docker_and_oci.html#apptainer-definition-file-vs-dockerfile) -- [GitHub Packages](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) +- [Apptainer User Guide (opens in new tab)](https://apptainer.org/docs/user/latest) +- [Definition File (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html) +- [Definition File vs Dockerfile (opens in new tab)](https://apptainer.org/docs/user/latest/docker_and_oci.html#apptainer-definition-file-vs-dockerfile) +- [GitHub Packages (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) diff --git a/content/notes/containers-for-hpc/building-docker.md b/content/notes/containers-for-hpc/building-docker.md index ed3554d5..a79176dd 100644 --- a/content/notes/containers-for-hpc/building-docker.md +++ b/content/notes/containers-for-hpc/building-docker.md @@ -1,14 +1,10 @@ --- -date : "2024-06-03T00:00:00" +date : "2024-6-03T00:00:00" title: Appendix 1 - Building Containers [Docker] toc: true -type: docs +type: book weight: 10 -menu: - containers-for-hpc: - parent: Software Containers for HPC - --- _Docker, Dockerfile, Docker Hub_ @@ -21,7 +17,7 @@ Starting from a simple Dockerfile, we will adopt best practices sequentially and 2. Install Docker - To follow along on your own computer, please install Docker Desktop and register for a free account on Docker Hub. Both can be found [here](https://www.docker.com/get-started). + To follow along on your own computer, please install Docker Desktop and register for a free account on Docker Hub. Both are available from [Docker Get Started (opens in new tab)](https://www.docker.com/get-started). After the installation, open a terminal ("cmd" on Windows) and make sure you can execute the command `docker run hello-world` successfully. @@ -38,7 +34,7 @@ Starting from a simple Dockerfile, we will adopt best practices sequentially and - Singularity - Users cannot build on Rivanna (needs `sudo` privilege) - Singularity Library/Hub (more limitations) - - Refer to [workshop](https://learning.rc.virginia.edu/workshops/singularity/) in Spring 2020 + - Refer to [workshop (opens in new tab)](https://learning.rc.virginia.edu/workshops/singularity/) in Spring 2020 ## Intro to Dockerfile: lolcow @@ -65,7 +61,7 @@ FROM ubuntu:22.04 - Doesn't have to be a bare OS - `python`, `continuumio/miniconda3`, `node`, `nvidia/cuda`, etc. -[Dockerfile reference](https://docs.docker.com/engine/reference/builder/#format) +[Dockerfile reference (opens in new tab)](https://docs.docker.com/engine/reference/builder/#format) ### Steps 2 & 3: Install software @@ -237,14 +233,14 @@ ENTRYPOINT fortune | cowsay | lolcat - You may need to specify extra packages - `fortune` itself provides the executable without the message database - `fortunes-min` contains the message database -- See [how Ubuntu reduced image size by 60%](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) +- See [how Ubuntu reduced image size by 60% (opens in new tab)](https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends) ### 3. Use a smaller base image For installation of common packages, you may consider Alpine. - BusyBox + package manager + musl libc (beware of compatibility issues) -- [Presentation](https://youtu.be/sIG2P9k6EjA) on Alpine Linux from DockerCon EU 17 +- [Presentation (opens in new tab)](https://youtu.be/sIG2P9k6EjA) on Alpine Linux from DockerCon EU 17 Look for `slim` variants (e.g. `debian:buster-slim`) of a base image, if any. @@ -281,7 +277,7 @@ $ docker images | grep lolcow | sort -nk 2 | awk '{print $1, $2, $NF}' |2 |Combination of previous two |53 | 26 | |3 |Alpine base image |161| 78 | -Reference: [_Best practices for writing Dockerfiles_](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +Reference: [_Best practices for writing Dockerfiles_ (opens in new tab)](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) ### Summary @@ -309,11 +305,11 @@ docker push /lolcow:latest ### Docker Hub interface -In your browser, go to `https://hub.docker.com/r//lolcow`. +In your browser, go to [Docker Hub for `/lolcow` (opens in new tab)](https://hub.docker.com/r//lolcow). - Overview: - Sync with GitHub to update `README.md`; or - - Use [docker-pushrm](https://github.com/christian-korneck/docker-pushrm) + - Use [docker-pushrm (opens in new tab)](https://github.com/christian-korneck/docker-pushrm) - Tags: - List all versions - View image history if Dockerfile not provided @@ -327,23 +323,23 @@ By now, we know how to write a simple Dockerfile to install software using the d ### Compiled language (C++) -https://github.com/lilab-bcb/cumulus_feature_barcoding +[cumulus_feature_barcoding repository (opens in new tab)](https://github.com/lilab-bcb/cumulus_feature_barcoding) Hints: - You do not have to start from a bare OS. Search for `gcc` on Docker Hub. - Install `build-essential` if you are starting from a bare OS. -- Version pinning - to choose a specific version, download from https://github.com/lilab-bcb/cumulus_feature_barcoding/releases (you will need `wget`). +- Version pinning - to choose a specific version, download from [cumulus_feature_barcoding releases (opens in new tab)](https://github.com/lilab-bcb/cumulus_feature_barcoding/releases) (you will need `wget`). ### Interpreted language (Python) -https://docs.qiime2.org/2022.8/install/native/#install-qiime-2-within-a-conda-environment +[QIIME 2 native install guide (opens in new tab)](https://docs.qiime2.org/2022.8/install/native/#install-qiime-2-within-a-conda-environment) Hints: - Click on "Linux" to get the URL for the yaml file. Download the yaml file in the same directory as your Dockerfile. - You do not have to start from a bare OS in your Dockerfile. Search for `miniconda3` on Docker Hub. -- (Recommended) There is a much faster dependency solver than conda - micromamba. See [here](https://micromamba-docker.readthedocs.io/en/latest/quick_start.html) for instructions. +- (Recommended) There is a much faster dependency solver than conda - micromamba. See the [Micromamba Docker quick start guide (opens in new tab)](https://micromamba-docker.readthedocs.io/en/latest/quick_start.html) for instructions. - Use the suggested `COPY` and `ENTRYPOINT` statements. -- After you're done, compare with the [official Dockerfile](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile) and image size. What is the biggest reason for the difference? +- After you're done, compare with the [official Dockerfile (opens in new tab)](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile) and image size. What is the biggest reason for the difference? ### General Remarks @@ -351,7 +347,7 @@ Hints: - If you encounter a Docker statement that you have not used before, first check the official documentation for best practices. - A comprehensive list of dependencies may be lacking. Some developers may not specify any at all. You will have to rely on a combination of experience, error message, and web search. (Most likely all of the above.) - Especially for Python packages, versions may be too permissive or too restrictive such that, in either case, future installation of the application will fail. (I have encountered both.) Tweak the versions until it works. -- The next step is "multi-stage build" which is covered in the [Minimal Containers](../minimal) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. +- The next step is "multi-stage build" which is covered in the [Minimal Containers](/workshops/minimal-containers) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. ## Clean Up @@ -375,8 +371,8 @@ If you build containers often, you can run out of disk space quickly. To clean u ## References -- [UVA Rivanna-Docker GitHub](https://github.com/uvarc/rivanna-docker) +- [UVA Rivanna-Docker GitHub (opens in new tab)](https://github.com/uvarc/rivanna-docker) - Dockerfiles by UVA Research Computing - - [Tips](https://github.com/uvarc/rivanna-docker/wiki/Tips) -- [_Best practices for writing Dockerfiles_](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) -- [Natanael Copa, _Small, Simple, and Secure: Alpine Linux under the Microscope_, DockerCon EU (2017)](https://youtu.be/sIG2P9k6EjA) + - [Tips (opens in new tab)](https://github.com/uvarc/rivanna-docker/wiki/Tips) +- [_Best practices for writing Dockerfiles_ (opens in new tab)](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +- [Natanael Copa, _Small, Simple, and Secure: Alpine Linux under the Microscope_, DockerCon EU (2017) (opens in new tab)](https://youtu.be/sIG2P9k6EjA) diff --git a/content/notes/containers-for-hpc/intro.md b/content/notes/containers-for-hpc/intro.md index 78348f66..99754a7c 100755 --- a/content/notes/containers-for-hpc/intro.md +++ b/content/notes/containers-for-hpc/intro.md @@ -1,17 +1,13 @@ --- title: Introduction to Software Containers toc: true -type: docs +type: book date: "2025-05-15T00:00:00" weight: 2 -menu: - containers-for-hpc: - parent: Software Containers for HPC - --- -"A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another." () +"A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another." ([Docker definition of a container (opens in new tab)](https://www.docker.com/resources/what-container)) ## Why use software containers? diff --git a/content/notes/containers-for-hpc/minimal.md b/content/notes/containers-for-hpc/minimal.md index d841d2d1..5e33e226 100644 --- a/content/notes/containers-for-hpc/minimal.md +++ b/content/notes/containers-for-hpc/minimal.md @@ -1,14 +1,10 @@ --- -date : "2026-03-09T00:00:00-05:00" +date : "2024-6-03T00:00:00-05:00" title: Appendix 2 - Minimal Containers [Docker] toc: true -type: docs +type: book weight: 11 -menu: - containers-for-hpc: - parent: Software Containers for HPC - --- The industry standard of restricting containers to just the application and its dependencies often results in better security and smaller size. See how the use of multi-stage builds and scratch/distroless base images can reduce the image size by as much as 99% in real applications. @@ -48,17 +44,17 @@ pip install --no-cache-dir ... ``` ### 3. Base image -- For OS and common tools (e.g. python/conda, GCC) start from [official image](https://docs.docker.com/docker-hub/official_images/) +- For OS and common tools (e.g. python/conda, GCC) start from [official image (opens in new tab)](https://docs.docker.com/docker-hub/official_images/) - Do not reinvent the wheel - Know which variant to use (e.g. `devel` vs `runtime`, `slim`) - Read their overview ## Exercise: QIIME 2 -QIIME 2 is a popular bioinformatics software. Can you suggest any potential improvement to the [Dockerfile](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile)? +QIIME 2 is a popular bioinformatics software. Can you suggest any potential improvement to the [Dockerfile (opens in new tab)](https://github.com/qiime2/vm-playbooks/blob/0fda9dce42802596756986e2f80c38437872c66e/docker/Dockerfile)?
Answer -With our [Dockerfile](https://github.com/uvarc/rivanna-docker/blob/master/qiime2/2020.8/Dockerfile), we managed to reduce the image size by half. +With our [Dockerfile (opens in new tab)](https://github.com/uvarc/rivanna-docker/blob/master/qiime2/2020.8/Dockerfile), we managed to reduce the image size by half.
@@ -84,7 +80,7 @@ The objective is to minimize image size without loss of functionality. What's ne - only install runtime dependencies (e.g. `cmake`, `go` not needed) - copy software from build stage -Reference: [_Use multi-stage builds_](https://docs.docker.com/develop/develop-images/multistage-build/) +Reference: [_Use multi-stage builds_ (opens in new tab)](https://docs.docker.com/develop/develop-images/multistage-build/) ### Multi-stage Dockerfile template @@ -109,11 +105,11 @@ ENTRYPOINT ["binary"] ## Exercise: LightGBM LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is an open source project by Microsoft. -1. Examine the [official Dockerfile](https://github.com/microsoft/LightGBM/blob/master/docker/gpu/dockerfile.gpu). Can you identify any problems? +1. Examine the [official Dockerfile (opens in new tab)](https://github.com/microsoft/LightGBM/blob/master/docker/gpu/dockerfile.gpu). Can you identify any problems?
Answer - - `nvidia/cuda:cudnn-devel` as [base image](https://hub.docker.com/r/nvidia/cuda/tags) (>1 GB) + - `nvidia/cuda:cudnn-devel` as [base image (opens in new tab)](https://hub.docker.com/r/nvidia/cuda/tags) (>1 GB) - Clean up in separate `RUN` statements
@@ -205,7 +201,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith
-3. Rewrite the Dockerfile using a multi-stage build based on [OpenCL](https://hub.docker.com/r/nvidia/cuda/tags). +3. Rewrite the Dockerfile using a multi-stage build based on [OpenCL (opens in new tab)](https://hub.docker.com/r/nvidia/cuda/tags). - Use `nvidia/opencl:devel` as the build stage - Remove everything related to CUDA since it is not relevant - Redefine the OpenCL environment variables as: @@ -216,7 +212,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith - Keep the same dependencies - Remember to clean up at the right place - Ignore the command under `# Add OpenCL ICD files for LightGBM` - - Instead of `mkdir foo && cd foo` use `WORKDIR foo` (see [documentation](https://docs.docker.com/engine/reference/builder/#workdir)) + - Instead of `mkdir foo && cd foo` use `WORKDIR foo` (see [documentation (opens in new tab)](https://docs.docker.com/engine/reference/builder/#workdir)) - Use the same command to install LightGBM, but replace the paths with the corresponding OpenCL environment variables - Add an `ENTRYPOINT` - Use `nvidia/opencl:runtime` as the production stage @@ -274,7 +270,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith
-4. Challenge: Verify that the two containers have the same performance on a GPU node. Follow the [tutorial example](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? +4. Challenge: Verify that the two containers have the same performance on Rivanna's GPU node. Follow the [tutorial example (opens in new tab)](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? --- @@ -294,12 +290,12 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith - Build production ### Example base images -- [Scratch](https://hub.docker.com/_/scratch) +- [Scratch (opens in new tab)](https://hub.docker.com/_/scratch) - Literally start from scratch! - A "no-op" in the Dockerfile, meaning no extra layer in image - Build other base images (beyond scope of this workshop) - Minimal image with a single application -- [Distroless](https://github.com/GoogleContainerTools/distroless) +- [Distroless (opens in new tab)](https://github.com/GoogleContainerTools/distroless) - Derived from Debian - Support for C/C++, Go, Rust, Java, Node.js, Python @@ -330,8 +326,8 @@ This exercise illustrates how we can cherry-pick files from the package manager
For your reference, the content of the packages can be found here: - - fortune-mod [package list](https://packages.ubuntu.com/xenial/all/fortune-mod/filelist) - - fortunes-min [package list](https://packages.ubuntu.com/xenial/all/fortunes-min/filelist) + - fortune-mod [package list (opens in new tab)](https://packages.ubuntu.com/xenial/all/fortune-mod/filelist) + - fortunes-min [package list (opens in new tab)](https://packages.ubuntu.com/xenial/all/fortunes-min/filelist)
@@ -443,21 +439,33 @@ The image size is merely **10.7 MB**, 99.5% smaller than what we started with. T -We submitted a [pull request](https://github.com/microsoft/LightGBM/pull/3408) that has been [merged](https://github.com/microsoft/LightGBM/tree/master/docker/gpu). +We submitted a [LightGBM GPU Docker pull request (opens in new tab)](https://github.com/microsoft/LightGBM/pull/3408) that has been [merged into the LightGBM Docker GPU directory (opens in new tab)](https://github.com/microsoft/LightGBM/tree/master/docker/gpu). + +--- + +## Example: TensorFlow distroless + +TensorFlow is a popular platform for machine learning. It is an open source project by Google. + +The TF 2.3 container that you used in the previous workshop is actually based on distroless, which is why you were not able to run `ls` inside the container. + +- [Dockerfile (opens in new tab)](https://github.com/uvarc/rivanna-docker/blob/master/tensorflow/2.3.0/Dockerfile.distroless) +- 18% image size reduction +- [TensorFlow build pull request (opens in new tab)](https://github.com/tensorflow/build/pull/13) approved and [merged into the tensorflow/build repository (opens in new tab)](https://github.com/tensorflow/build) --- ## Dynamic vs Static Linking -The above procedure, while impressive, may be tedious for the average user. All the examples so far are based on [dynamic linking](https://en.wikipedia.org/wiki/Dynamic_linker), where the shared libraries of an executable are stored separately. If you are compiling code from source, you may choose to build a static binary (e.g. `-static` in GCC) so that all the necessary libraries are built into the binary. +The above procedure, while impressive, may be tedious for the average user. All the examples so far are based on [dynamic linking (opens in new tab)](https://en.wikipedia.org/wiki/Dynamic_linker), where the shared libraries of an executable are stored separately. If you are compiling code from source, you may choose to build a static binary (e.g. `-static` in GCC) so that all the necessary libraries are built into the binary. ## Exercise: Linking against OpenBLAS -OpenBLAS is a linear algebra library. The code in this exercise is taken from its [user manual](https://github.com/xianyi/OpenBLAS/wiki/User-Manual#code-examples). It is based on `dgemm` which performs the matrix operation: +OpenBLAS is a linear algebra library. The code in this exercise is taken from its [user manual (opens in new tab)](https://github.com/xianyi/OpenBLAS/wiki/User-Manual#code-examples). It is based on `dgemm` which performs the matrix operation: $$ \alpha A B + \beta C, $$ -where $A_{mk}, B_{kn}, C_{mn}$ are matrices and $\alpha, \beta$ are constants. For details please visit the [Intel tutorial page](https://software.intel.com/content/www/us/en/develop/documentation/mkl-tutorial-c/top/multiplying-matrices-using-dgemm.html). +where $A_{mk}, B_{kn}, C_{mn}$ are matrices and $\alpha, \beta$ are constants. For details please visit the [Intel tutorial page (opens in new tab)](https://software.intel.com/content/www/us/en/develop/documentation/mkl-tutorial-c/top/multiplying-matrices-using-dgemm.html). 1. Select an appropriate base image. (Hint: You will be compiling C++ code.) @@ -568,7 +576,7 @@ where $A_{mk}, B_{kn}, C_{mn}$ are matrices and $\alpha, \beta$ are constants. F This exercise illustrates that it is easier to build a minimal container of a static binary. ## Exercise: Linking against LibTorch -LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing a Basic Application"](https://pytorch.org/tutorials/advanced/cpp_frontend.html#writing-a-basic-application) section of the PyTorch tutorial. +LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing a Basic Application" (opens in new tab)](https://pytorch.org/tutorials/advanced/cpp_frontend.html#writing-a-basic-application) section of the PyTorch tutorial. 1. Select an appropriate base image. (Hint: You will be compiling C++ code.) @@ -576,7 +584,7 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing - Build tools: `build-essential cmake` - Download and decompress: `wget ca-certificates unzip` -1. Find the download link for LibTorch under the "Install PyTorch" section at https://pytorch.org/. Select "None" for CUDA. Download the file to `/opt` in the image. Hints: +1. Find the download link for LibTorch under the "Install PyTorch" section at [pytorch.org (opens in new tab)](https://pytorch.org/). Select "None" for CUDA. Download the file to `/opt` in the image. Hints: - In your `wget` command, you may want to rename the output file using `-O libtorch.zip`. - Remember to decompress. @@ -679,11 +687,13 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing 1. Challenge: The above image cannot make use of GPU. Build an image for GPU. Hints: - You do not need a physical GPU to build an image for GPU. - - Use `nvidia/cuda` as the base image. Read their overview page on Docker Hub to decide which flavor to use. + - Pick a `nvidia/cuda` base image. Read their overview page on Docker Hub to decide which flavor to use. - Choose a CUDA version to get the download link for LibTorch on the PyTorch webpage.
+1. Challenge: Can you build `dcgan` on Rivanna without using a container? Why (not)? + 1. Challenge: Can you build a static binary of `dcgan`? Why (not)? --- @@ -703,10 +713,10 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing ## References -- [UVA Rivanna-Docker GitHub](https://github.com/uvarc/rivanna-docker) +- [UVA Rivanna-Docker GitHub (opens in new tab)](https://github.com/uvarc/rivanna-docker) - Dockerfiles by UVA Research Computing - - [Tips](https://github.com/uvarc/rivanna-docker/wiki/Tips) -- [_Best practices for writing Dockerfiles_](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) -- [_Use multi-stage builds_](https://docs.docker.com/develop/develop-images/multistage-build/) -- [Google Distroless GitHub](https://github.com/GoogleContainerTools/distroless) -- [Matthew Moore, _Distroless Docker: Containerizing Apps, not VMs_, swampUP (2017)](https://youtu.be/lviLZFciDv4) + - [Tips (opens in new tab)](https://github.com/uvarc/rivanna-docker/wiki/Tips) +- [_Best practices for writing Dockerfiles_ (opens in new tab)](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) +- [_Use multi-stage builds_ (opens in new tab)](https://docs.docker.com/develop/develop-images/multistage-build/) +- [Google Distroless GitHub (opens in new tab)](https://github.com/GoogleContainerTools/distroless) +- [Matthew Moore, _Distroless Docker: Containerizing Apps, not VMs_, swampUP (2017) (opens in new tab)](https://youtu.be/lviLZFciDv4) diff --git a/content/notes/containers-for-hpc/using.md b/content/notes/containers-for-hpc/using.md index 869a6414..2bc19673 100755 --- a/content/notes/containers-for-hpc/using.md +++ b/content/notes/containers-for-hpc/using.md @@ -1,14 +1,10 @@ --- title: Using Containers on HPC [Apptainer] toc: true -type: docs -date: "2026-03-16T00:00:00" +type: book +date: "2025-05-13T00:00:00" weight: 3 -menu: - containers-for-hpc: - parent: Software Containers for HPC - --- Log on to our HPC cluster @@ -18,7 +14,7 @@ Log on to our HPC cluster - Make sure you have a few GBs of free space - Run `allocations` - Check if you have `hpc_training` -- Request an interactive job +- Request an interative job {{< code-snippet >}}ijob -A hpc_training -p interactive -c 1 -t 2:0:0{{< /code-snippet >}} - Run `module load apptainer` @@ -34,7 +30,7 @@ To download a container hosted on a registry, use the `pull` command. Docker ima - `` (Unified resource identifiers) - `[library|docker|shub]://[/][:] ` - - Default prefix: `library` ([Singularity Library](https://cloud.sylabs.io/library)) + - Default prefix: `library` ([Singularity Library (opens in new tab)](https://cloud.sylabs.io/library)) - `user`: optional; may be empty (e.g. `apptainer pull ubuntu`) - `tag`: optional; default: `latest` - `` (Singularity image format) @@ -56,9 +52,9 @@ Inspect an image before running it via `inspect`. ```bash $ apptainer inspect lolcow_latest.sif org.label-schema.build-arch: amd64 -org.label-schema.build-date: Monday_16_March_2026_12:35:7_EDT +org.label-schema.build-date: Wednesday_14_May_2025_10:20:6_EDT org.label-schema.schema-version: 1.0 -org.label-schema.usage.apptainer.version: 1.4.5 +org.label-schema.usage.apptainer.version: 1.3.4 org.label-schema.usage.singularity.deffile.bootstrap: docker org.label-schema.usage.singularity.deffile.from: rsdmse/lolcow ``` @@ -67,29 +63,15 @@ org.label-schema.usage.singularity.deffile.from: rsdmse/lolcow This is the default command of the container. (Docker `ENTRYPOINT` is preserved.) -`apptainer inspect -r/--runscript ` +`apptainer inspect --runscript ` ```bash -$ apptainer inspect -r lolcow_latest.sif +$ apptainer inspect --runscript lolcow_latest.sif #!/bin/sh OCI_ENTRYPOINT='"/bin/sh" "-c" "fortune | cowsay | lolcat"' ... ``` -#### Inspect definition file - -The definition file is a recipe that defines how the container is built. We will learn more about it in the next workshop. - -`apptainer inspect -d/--deffile ` - -```bash -$ apptainer inspect -d lolcow_latest.sif -bootstrap: docker -from: rsdmse/lolcow -``` - -(Not very interesting for containers converted from Docker. It cannot show how the _base image_ was built.) - ### Run There are three ways to run a container: `run`, `shell`, `exec`. @@ -264,7 +246,7 @@ sbatch tensorflow-2.17.0.slurm #### What does `--nv` do? -See [Apptainer GPU user guide](https://apptainer.org/user-docs/master/gpu.html#nvidia-gpus-cuda-standard) +See [Apptainer GPU user guide (opens in new tab)](https://apptainer.org/user-docs/master/gpu.html#nvidia-gpus-cuda-standard) ```bash $ apptainer shell $CONTAINERDIR/tensorflow-2.17.0.sif @@ -289,7 +271,7 @@ Suppose you need to use TensorFlow 2.19.0 on JupyterLab. First, note we do not h module spider tensorflow ``` -Go to [TensorFlow's Docker Hub page](https://hub.docker.com/r/tensorflow/tensorflow) and search for the tag (i.e. version). You'll want to use one that has the `-gpu-jupyter` suffix. Pull the container in your account. +Go to [TensorFlow's Docker Hub page (opens in new tab)](https://hub.docker.com/r/tensorflow/tensorflow) and search for the tag (i.e. version). You'll want to use one that has the `-gpu-jupyter` suffix. Pull the container in your account. ### Installation @@ -348,7 +330,7 @@ jkrollout /path/to/sif "Tensorflow 2.19" gpu ### Test your new kernel -- Go to https://ood.hpc.virginia.edu +- Go to [Open OnDemand at UVA (opens in new tab)](https://ood.hpc.virginia.edu) - Select JupyterLab - Partition: GPU - Work Directory: (location of your `mnist_example.ipynb`) @@ -366,6 +348,6 @@ rm -rf ~/.local/share/jupyter/kernels/tensorflow-2.19.0 ## References -- [Apptainer User Guide](https://apptainer.org/docs/user/latest/) - - [Overview](https://apptainer.org/docs/user/latest/quick_start.html) - - [Bind Path and Mounts](https://apptainer.org/docs/user/latest/bind_paths_and_mounts.html) +- [Apptainer User Guide (opens in new tab)](https://apptainer.org/docs/user/latest/) + - [Apptainer quick start overview (opens in new tab)](https://apptainer.org/docs/user/latest/quick_start.html) + - [Bind Path and Mounts (opens in new tab)](https://apptainer.org/docs/user/latest/bind_paths_and_mounts.html) From 96d759518594d37a7ed9875bb07f4fd44bab8819 Mon Sep 17 00:00:00 2001 From: Aaditi Rai Date: Wed, 18 Mar 2026 11:21:43 -0400 Subject: [PATCH 2/5] Fix to _index.md, reverting it back to the original --- content/notes/containers-for-hpc/_index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/content/notes/containers-for-hpc/_index.md b/content/notes/containers-for-hpc/_index.md index 61015558..a0041dd2 100644 --- a/content/notes/containers-for-hpc/_index.md +++ b/content/notes/containers-for-hpc/_index.md @@ -5,14 +5,17 @@ authors: [rs] categories: ["Containers","HPC"] tags: [Containers, HPC] toc: true -type: book +type: docs date: "2025-05-15T00:00:00" weight: 1 +menu: + containers-for-hpc: + name: Software Containers for HPC --- ## Overview -This short course is an introduction to using and building software containers, structured as follows: +This tutorial is an introduction to using and building software containers, structured as follows: - Introduction to Software Containers - Using Containers on HPC [Apptainer] From 35288c6bb75fcb147c6e1688aecc7257b1251816 Mon Sep 17 00:00:00 2001 From: Aaditi Rai Date: Wed, 18 Mar 2026 11:32:26 -0400 Subject: [PATCH 3/5] CHnaging types to docs for the content in notes for workshop --- content/notes/containers-for-hpc/building-apptainer.md | 4 ++-- content/notes/containers-for-hpc/building-docker.md | 2 +- content/notes/containers-for-hpc/intro.md | 2 +- content/notes/containers-for-hpc/minimal.md | 2 +- content/notes/containers-for-hpc/using.md | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/content/notes/containers-for-hpc/building-apptainer.md b/content/notes/containers-for-hpc/building-apptainer.md index a0b348f6..3bef5fc8 100644 --- a/content/notes/containers-for-hpc/building-apptainer.md +++ b/content/notes/containers-for-hpc/building-apptainer.md @@ -2,7 +2,7 @@ date: "2025-06-04T00:00:00" title: "Building Containers on HPC [Apptainer]" toc: true -type: book +type: docs weight: 4 --- @@ -522,4 +522,4 @@ This is beyond the scope of the workshop, but you are welcome to browse the Appt - [Apptainer User Guide (opens in new tab)](https://apptainer.org/docs/user/latest) - [Definition File (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html) - [Definition File vs Dockerfile (opens in new tab)](https://apptainer.org/docs/user/latest/docker_and_oci.html#apptainer-definition-file-vs-dockerfile) -- [GitHub Packages (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) +- [GitHub Packages (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) \ No newline at end of file diff --git a/content/notes/containers-for-hpc/building-docker.md b/content/notes/containers-for-hpc/building-docker.md index a79176dd..1d2c6122 100644 --- a/content/notes/containers-for-hpc/building-docker.md +++ b/content/notes/containers-for-hpc/building-docker.md @@ -2,7 +2,7 @@ date : "2024-6-03T00:00:00" title: Appendix 1 - Building Containers [Docker] toc: true -type: book +type: docs weight: 10 --- diff --git a/content/notes/containers-for-hpc/intro.md b/content/notes/containers-for-hpc/intro.md index 99754a7c..573abaf6 100755 --- a/content/notes/containers-for-hpc/intro.md +++ b/content/notes/containers-for-hpc/intro.md @@ -1,7 +1,7 @@ --- title: Introduction to Software Containers toc: true -type: book +type: docs date: "2025-05-15T00:00:00" weight: 2 diff --git a/content/notes/containers-for-hpc/minimal.md b/content/notes/containers-for-hpc/minimal.md index 5e33e226..87fb6796 100644 --- a/content/notes/containers-for-hpc/minimal.md +++ b/content/notes/containers-for-hpc/minimal.md @@ -2,7 +2,7 @@ date : "2024-6-03T00:00:00-05:00" title: Appendix 2 - Minimal Containers [Docker] toc: true -type: book +type: docs weight: 11 --- diff --git a/content/notes/containers-for-hpc/using.md b/content/notes/containers-for-hpc/using.md index 2bc19673..9a644ecb 100755 --- a/content/notes/containers-for-hpc/using.md +++ b/content/notes/containers-for-hpc/using.md @@ -1,7 +1,7 @@ --- title: Using Containers on HPC [Apptainer] toc: true -type: book +type: docs date: "2025-05-13T00:00:00" weight: 3 From 72429672ce46dfb3c42fd0ae863512b1bb1dc439 Mon Sep 17 00:00:00 2001 From: Katherine Holcomb Date: Wed, 18 Mar 2026 14:34:11 -0400 Subject: [PATCH 4/5] Fixed some regressions --- content/notes/containers-for-hpc/_index.md | 3 --- .../containers-for-hpc/building-apptainer.md | 6 +++++- .../containers-for-hpc/building-docker.md | 4 ++++ content/notes/containers-for-hpc/intro.md | 3 +++ content/notes/containers-for-hpc/minimal.md | 4 ++++ content/notes/containers-for-hpc/using.md | 4 ++++ content/notes/containers/_index.md | 21 ------------------- content/notes/containers/overview-purpose.md | 21 ------------------- content/notes/containers/overview-services.md | 20 ------------------ content/notes/containers/overview-vms.md | 18 ---------------- 10 files changed, 20 insertions(+), 84 deletions(-) delete mode 100755 content/notes/containers/_index.md delete mode 100755 content/notes/containers/overview-purpose.md delete mode 100755 content/notes/containers/overview-services.md delete mode 100755 content/notes/containers/overview-vms.md diff --git a/content/notes/containers-for-hpc/_index.md b/content/notes/containers-for-hpc/_index.md index a0041dd2..2dcd0a32 100644 --- a/content/notes/containers-for-hpc/_index.md +++ b/content/notes/containers-for-hpc/_index.md @@ -1,9 +1,6 @@ --- title: "Software Containers for HPC" -summary: "An Introduction to using and building software containers." authors: [rs] -categories: ["Containers","HPC"] -tags: [Containers, HPC] toc: true type: docs date: "2025-05-15T00:00:00" diff --git a/content/notes/containers-for-hpc/building-apptainer.md b/content/notes/containers-for-hpc/building-apptainer.md index 3bef5fc8..8a492c02 100644 --- a/content/notes/containers-for-hpc/building-apptainer.md +++ b/content/notes/containers-for-hpc/building-apptainer.md @@ -5,6 +5,10 @@ toc: true type: docs weight: 4 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- ## Introduction @@ -522,4 +526,4 @@ This is beyond the scope of the workshop, but you are welcome to browse the Appt - [Apptainer User Guide (opens in new tab)](https://apptainer.org/docs/user/latest) - [Definition File (opens in new tab)](https://apptainer.org/docs/user/latest/definition_files.html) - [Definition File vs Dockerfile (opens in new tab)](https://apptainer.org/docs/user/latest/docker_and_oci.html#apptainer-definition-file-vs-dockerfile) -- [GitHub Packages (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) \ No newline at end of file +- [GitHub Packages (opens in new tab)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) diff --git a/content/notes/containers-for-hpc/building-docker.md b/content/notes/containers-for-hpc/building-docker.md index 1d2c6122..5d5d002d 100644 --- a/content/notes/containers-for-hpc/building-docker.md +++ b/content/notes/containers-for-hpc/building-docker.md @@ -5,6 +5,10 @@ toc: true type: docs weight: 10 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- _Docker, Dockerfile, Docker Hub_ diff --git a/content/notes/containers-for-hpc/intro.md b/content/notes/containers-for-hpc/intro.md index 573abaf6..5c031131 100755 --- a/content/notes/containers-for-hpc/intro.md +++ b/content/notes/containers-for-hpc/intro.md @@ -5,6 +5,9 @@ type: docs date: "2025-05-15T00:00:00" weight: 2 +menu: + containers-for-hpc: + parent: Software Containers for HPC --- "A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another." ([Docker definition of a container (opens in new tab)](https://www.docker.com/resources/what-container)) diff --git a/content/notes/containers-for-hpc/minimal.md b/content/notes/containers-for-hpc/minimal.md index 87fb6796..97a2b909 100644 --- a/content/notes/containers-for-hpc/minimal.md +++ b/content/notes/containers-for-hpc/minimal.md @@ -5,6 +5,10 @@ toc: true type: docs weight: 11 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- The industry standard of restricting containers to just the application and its dependencies often results in better security and smaller size. See how the use of multi-stage builds and scratch/distroless base images can reduce the image size by as much as 99% in real applications. diff --git a/content/notes/containers-for-hpc/using.md b/content/notes/containers-for-hpc/using.md index 9a644ecb..05c442ab 100755 --- a/content/notes/containers-for-hpc/using.md +++ b/content/notes/containers-for-hpc/using.md @@ -5,6 +5,10 @@ type: docs date: "2025-05-13T00:00:00" weight: 3 +menu: + containers-for-hpc: + parent: Software Containers for HPC + --- Log on to our HPC cluster diff --git a/content/notes/containers/_index.md b/content/notes/containers/_index.md deleted file mode 100755 index c5fef2bd..00000000 --- a/content/notes/containers/_index.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -# Course title, summary, and position. -title: "Introduction to Building and Deploying Containers" -authors: [cag] - -# Page metadata. -date: "2023-05-01T00:00:00" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. - -# Add menu entry to sidebar. -menu: - containers: - name: Containers - weight: 5 ---- - -You've developed an app and written the paper. Now it's time to deploy the app so the world (and the reviewers) can see how awesome it is. This is Part 1 of a two-part workshop that will cover how to deploy web apps for publication. In Part 1 we will go over how to containerize our apps with Docker and maintain them with GitHub. - -docker logo img github logo image diff --git a/content/notes/containers/overview-purpose.md b/content/notes/containers/overview-purpose.md deleted file mode 100755 index b007d7f2..00000000 --- a/content/notes/containers/overview-purpose.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Why Use Containers? -date: "2023-05-01:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 20 -date: "2023-05-01T00:00:00Z" -menu: - containers: - parent: Containers ---- - -Have you ever tried using new code or software from an exciting paper you just read, only to end up spending hours figuring out which versions of the dependencies work on your own machine? Containers eliminate that issue altogether! - -A container is a single unit of software that contains all the packages and code you need to run an application. Sometimes that application is as small as a single function (like printing 'Hello World!'), and sometimes that application is an entire web app. A container will always run the same, regardless of the host system it runs on--making it the perfect solution for sharing reproducible code. - -There are several container technologies out there, but the big ones are Docker and Singularity. Docker is what you will encounter most often in the wild. Singularity (now called Apptainer) is used on HPC systems where most users don't have root access. - -docker logo image -apptainer logo image \ No newline at end of file diff --git a/content/notes/containers/overview-services.md b/content/notes/containers/overview-services.md deleted file mode 100755 index dc65195f..00000000 --- a/content/notes/containers/overview-services.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Container Services -date: "2023-05-01:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 40 -date: "2023-05-01T00:00:00Z" -menu: - containers: - parent: Containers ---- - -Container-based architecture, also known as "microservices," is an approach to designing and running applications as a distributed set of components or layers. Such applications are typically run within containers, made popular in the last few years by Docker. - -Containers are portable, efficient, reusable, and contain code and any dependencies in a single package. Containerized services typically run a single process, rather than an entire stack within the same environment. This allows developers to replace, scale, or troubleshoot portions of their entire application at a time. - -Diagram of multiple containers running separate services in a cluster. - -Research Computing runs microservices in a clustered orchestration environment that automates the deployment and management of many containers easy and scalable. This cluster has >1000 cores and ~1TB of memory allocated to running containerized services. It also has over 300TB of cluster storage and can attach to project and value storage. \ No newline at end of file diff --git a/content/notes/containers/overview-vms.md b/content/notes/containers/overview-vms.md deleted file mode 100755 index 23aa4c74..00000000 --- a/content/notes/containers/overview-vms.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: Containers vs VMs -date: "2023-05-01:00:00Z" -draft: false # Is this a draft? true/false -toc: false # Show table of contents? true/false -type: docs # Do not modify. -weight: 30 -date: "2023-05-01T00:00:00Z" -menu: - containers: - parent: Containers ---- - -You may be familiar with the virtual machines (VMs), which accomplish the same goal as containers: to create a reproducible and shareable compute environment. The big difference between VMs and containers is that VMs provide their own guest OS, whereas containers don't require an OS and run on the Docker Engine and share the host's OS kernel. - -The size of Docker images is usually on the order of tens of MBs, while VMs can be several tens of GBs large. - -Side-by-side comparison of containerized applications (Docker on a host OS) and virtual machines (apps running on guest OSs over a hypervisor). \ No newline at end of file From 0613ee014c198da5705c0a97bd9e70d0bc4d4553 Mon Sep 17 00:00:00 2001 From: rsdmse Date: Wed, 18 Mar 2026 14:46:03 -0400 Subject: [PATCH 5/5] redo edits --- .../containers-for-hpc/building-docker.md | 4 +-- content/notes/containers-for-hpc/minimal.md | 20 +++----------- content/notes/containers-for-hpc/using.md | 26 ++++++++++++++----- 3 files changed, 25 insertions(+), 25 deletions(-) diff --git a/content/notes/containers-for-hpc/building-docker.md b/content/notes/containers-for-hpc/building-docker.md index 5d5d002d..23191b71 100644 --- a/content/notes/containers-for-hpc/building-docker.md +++ b/content/notes/containers-for-hpc/building-docker.md @@ -1,5 +1,5 @@ --- -date : "2024-6-03T00:00:00" +date : "2024-06-03T00:00:00" title: Appendix 1 - Building Containers [Docker] toc: true type: docs @@ -351,7 +351,7 @@ Hints: - If you encounter a Docker statement that you have not used before, first check the official documentation for best practices. - A comprehensive list of dependencies may be lacking. Some developers may not specify any at all. You will have to rely on a combination of experience, error message, and web search. (Most likely all of the above.) - Especially for Python packages, versions may be too permissive or too restrictive such that, in either case, future installation of the application will fail. (I have encountered both.) Tweak the versions until it works. -- The next step is "multi-stage build" which is covered in the [Minimal Containers](/workshops/minimal-containers) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. +- The next step is "multi-stage build" which is covered in the [Minimal Containers](/notes/containers-for-hpc/minimal) workshop. There you will learn how to distinguish between buildtime versus runtime dependencies and separate them out. ## Clean Up diff --git a/content/notes/containers-for-hpc/minimal.md b/content/notes/containers-for-hpc/minimal.md index 97a2b909..b9a47a7f 100644 --- a/content/notes/containers-for-hpc/minimal.md +++ b/content/notes/containers-for-hpc/minimal.md @@ -1,5 +1,5 @@ --- -date : "2024-6-03T00:00:00-05:00" +date : "2026-03-09T00:00:00-05:00" title: Appendix 2 - Minimal Containers [Docker] toc: true type: docs @@ -274,7 +274,7 @@ LightGBM is a gradient boosting framework that uses tree based learning algorith
-4. Challenge: Verify that the two containers have the same performance on Rivanna's GPU node. Follow the [tutorial example (opens in new tab)](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? +4. Challenge: Verify that the two containers have the same performance on a GPU node. Follow the [tutorial example (opens in new tab)](https://lightgbm.readthedocs.io/en/latest/GPU-Tutorial.html#dataset-preparation). Run the same job without using GPU. How much faster is it with GPU? --- @@ -447,18 +447,6 @@ We submitted a [LightGBM GPU Docker pull request (opens in new tab)](https://git --- -## Example: TensorFlow distroless - -TensorFlow is a popular platform for machine learning. It is an open source project by Google. - -The TF 2.3 container that you used in the previous workshop is actually based on distroless, which is why you were not able to run `ls` inside the container. - -- [Dockerfile (opens in new tab)](https://github.com/uvarc/rivanna-docker/blob/master/tensorflow/2.3.0/Dockerfile.distroless) -- 18% image size reduction -- [TensorFlow build pull request (opens in new tab)](https://github.com/tensorflow/build/pull/13) approved and [merged into the tensorflow/build repository (opens in new tab)](https://github.com/tensorflow/build) - ---- - ## Dynamic vs Static Linking The above procedure, while impressive, may be tedious for the average user. All the examples so far are based on [dynamic linking (opens in new tab)](https://en.wikipedia.org/wiki/Dynamic_linker), where the shared libraries of an executable are stored separately. If you are compiling code from source, you may choose to build a static binary (e.g. `-static` in GCC) so that all the necessary libraries are built into the binary. @@ -691,13 +679,11 @@ LibTorch is the C++ frontend of PyTorch. This exercise is based on the ["Writing 1. Challenge: The above image cannot make use of GPU. Build an image for GPU. Hints: - You do not need a physical GPU to build an image for GPU. - - Pick a `nvidia/cuda` base image. Read their overview page on Docker Hub to decide which flavor to use. + - Use `nvidia/cuda` as the base image. Read their overview page on Docker Hub to decide which flavor to use. - Choose a CUDA version to get the download link for LibTorch on the PyTorch webpage.
-1. Challenge: Can you build `dcgan` on Rivanna without using a container? Why (not)? - 1. Challenge: Can you build a static binary of `dcgan`? Why (not)? --- diff --git a/content/notes/containers-for-hpc/using.md b/content/notes/containers-for-hpc/using.md index 05c442ab..13b73770 100755 --- a/content/notes/containers-for-hpc/using.md +++ b/content/notes/containers-for-hpc/using.md @@ -2,7 +2,7 @@ title: Using Containers on HPC [Apptainer] toc: true type: docs -date: "2025-05-13T00:00:00" +date: "2026-03-16T00:00:00" weight: 3 menu: @@ -18,7 +18,7 @@ Log on to our HPC cluster - Make sure you have a few GBs of free space - Run `allocations` - Check if you have `hpc_training` -- Request an interative job +- Request an interactive job {{< code-snippet >}}ijob -A hpc_training -p interactive -c 1 -t 2:0:0{{< /code-snippet >}} - Run `module load apptainer` @@ -56,9 +56,9 @@ Inspect an image before running it via `inspect`. ```bash $ apptainer inspect lolcow_latest.sif org.label-schema.build-arch: amd64 -org.label-schema.build-date: Wednesday_14_May_2025_10:20:6_EDT +org.label-schema.build-date: Monday_16_March_2026_12:35:7_EDT org.label-schema.schema-version: 1.0 -org.label-schema.usage.apptainer.version: 1.3.4 +org.label-schema.usage.apptainer.version: 1.4.5 org.label-schema.usage.singularity.deffile.bootstrap: docker org.label-schema.usage.singularity.deffile.from: rsdmse/lolcow ``` @@ -67,15 +67,29 @@ org.label-schema.usage.singularity.deffile.from: rsdmse/lolcow This is the default command of the container. (Docker `ENTRYPOINT` is preserved.) -`apptainer inspect --runscript ` +`apptainer inspect -r/--runscript ` ```bash -$ apptainer inspect --runscript lolcow_latest.sif +$ apptainer inspect -r lolcow_latest.sif #!/bin/sh OCI_ENTRYPOINT='"/bin/sh" "-c" "fortune | cowsay | lolcat"' ... ``` +#### Inspect definition file + +The definition file is a recipe that defines how the container is built. We will learn more about it in the next workshop. + +`apptainer inspect -d/--deffile ` + +```bash +$ apptainer inspect -d lolcow_latest.sif +bootstrap: docker +from: rsdmse/lolcow +``` + +(Not very interesting for containers converted from Docker. It cannot show how the _base image_ was built.) + ### Run There are three ways to run a container: `run`, `shell`, `exec`.