ICRN Kernel Manager

What is the ICRN Kernel Manager?

The Illinois Computes Research Notebook (ICRN) enables students and researchers to access computing at scale via an easily accessible web interface. But, many scientific domains rely on a wide array of complex packages for R and Python which are not easy to install. It is common for new users of compute systems to spend hours attempting to configure their environments.

The ICRN Kernel Manager aims to eliminate this barrier by providing a simple command-line interface and web-based catalog for managing pre-configured kernel environments.

Architecture Overview

The ICRN Kernel Manager consists of three main components:

1. Command-Line Interface (icrn_manager): The primary tool for users to discover, download, and activate kernel environments
2. Kernel Indexer: A containerized service that indexes kernel repositories and generates JSON manifest files containing package information
3. Web Interface: A FastAPI + Nginx web server that provides a browser-based interface for exploring available kernels and searching for packages

Component Integration

Kernel Repository
    ↓ (read-write mount)
Kernel Indexer (CronJob) → Generates collated_manifests.json & package_index.json
    ↓ (JSON files)
Web Server (reads JSON files) → Serves via REST API
    ↓
CLI Tool & Web UI → Users discover and manage kernels

User Installation

Usage

The ICRN Kernel Manager has all familiar operations for listing, getting, and using entites from a catalog or repository of kernel packages organized by language:

Available Packages in the Catalog

./icrn_manager kernels available

This command interrogates the central catalog of available packages, and lists them out, organized by language with their various versions.

You can check out more than one version of each package set at a time, but you can only use one package set at any time.

Available kernels in ICRN catalog (/u/hdpriest/icrn_temp_repository/icrn_kernel_catalog.json):
Language	Kernel	Version
R	cowsay	1.0
R	mixRF	1.0
R	pecan	1.9
R	vctrs	1.0
Python	numpy	1.20

Alias:

./icrn_manager kernels avail

Use a package set

./icrn_manager kernels use <language> <kernel> <version>

Example:

./icrn_manager kernels use R cowsay 1.0
./icrn_manager kernels use Python numpy 1.24.0

Switch to a different set of packages

./icrn_manager kernels use R pecan 1.9

Stop using package sets

./icrn_manager kernels use R none

Kernel Indexer

The kernel indexer is a containerized service that indexes kernel repositories and generates JSON manifest files. It scans kernel directories, extracts package information, and creates two key files:

• collated_manifests.json: A kernel-centric index listing all available kernels with their packages
• package_index.json: A package-centric index showing which kernels contain each package

Building the Kernel Indexer

docker build -t icrn-kernel-indexer:latest -f kernel-indexer/Dockerfile .

Running the Kernel Indexer

The indexer requires read-write access to the kernel repository:

docker run --rm \
  -v /path/to/kernel/repository:/sw/icrn/jupyter/icrn_ncsa_resources/Kernels \
  -e KERNEL_ROOT=/sw/icrn/jupyter/icrn_ncsa_resources/Kernels \
  icrn-kernel-indexer:latest

Kubernetes Deployment

The kernel indexer is designed to run as a Kubernetes CronJob. See kernel-indexer/README.md for detailed deployment instructions.

For more information, see the Kernel Indexer documentation and design document.

Web Interface

The ICRN Kernel Manager includes a web-based interface for browsing available kernels and searching for packages. The web interface reads the JSON files generated by the kernel indexer and provides an intuitive way to explore the kernel catalog without using the command line.

Prerequisites

The web interface requires the JSON files generated by the kernel indexer:

• collated_manifests.json
• package_index.json

These files should be located in the data directory or mounted as volumes.

Starting the Web Interface with Docker

1. Build the Docker image:

   docker build -t icrn-kernel-webserver:latest -f web/Dockerfile web/

2. Run the container with kernel data:

   docker run -d -p 8080:80 \
     -v /path/to/kernel/repository:/app/data:ro \
     -e COLLATED_MANIFESTS_PATH=/app/data/collated_manifests.json \
     -e PACKAGE_INDEX_PATH=/app/data/package_index.json \
     --name icrn-web icrn-kernel-webserver:latest

   Or with example data:

   docker run -d -p 8080:80 --name icrn-web icrn-kernel-webserver:latest
   docker cp web/examples/collated_manifests.json icrn-web:/app/data/
   docker cp web/examples/package_index.json icrn-web:/app/data/

3. Access the web interface:

   Open your web browser and navigate to:

   http://localhost:8080
   

Web Interface Features

The web interface provides two main views:

• View Kernels: Browse available kernels organized by language, view kernel details, and see packages included in each kernel. You can filter and sort packages, and copy commands to get or use specific kernels.

• View Packages: Search for packages by name to see which kernels contain them. Results show the language, kernel name, kernel version, and package version for each match.

API Endpoints

The web interface exposes a REST API:

• GET /: API information and status
• GET /health: Health check endpoint
• GET /api/languages: Get list of available languages
• GET /api/kernels/{language}: Get all kernels for a specific language
• GET /api/kernel/{language}/{kernel_name}/{version}: Get specific kernel details
• GET /api/manifest/{language}/{kernel_name}/{version}: Get package manifest for a kernel
• GET /api/package/{package_name}: Get all kernels containing a package
• GET /api/packages/search: Search for packages by name
• POST /api/refresh: Manually trigger data refresh (reloads JSON files from disk)

For more information, see the Web Interface documentation.

Containerized Deployment

The ICRN Kernel Manager components are containerized for easy deployment:

Available Container Images

1. Kernel Indexer: icrn-kernel-indexer

   • Automatically builds and pushes to Docker Hub on commits to main/develop branches
   • Designed for Kubernetes CronJob deployment
   • See kernel-indexer/README.md for details

2. Web Server: icrn-kernel-webserver

   • Automatically builds and pushes to Docker Hub on commits to main/develop branches
   • FastAPI backend with Nginx reverse proxy
   • See web/README.md for details

3. Apptainer/Singularity Support: The kernel indexer also supports Apptainer/Singularity containers

   • See kernel-indexer/APPTAINER_EXAMPLE.md for usage

Continuous Integration

GitHub Actions workflows automatically build and push Docker images:

• docker-build-indexer.yml: Builds and pushes kernel indexer container
• docker-build-webserver.yml: Builds and pushes web server container
• docker-build.yml: Builds the main CLI tool container

All workflows trigger on pushes and pull requests to main and develop branches.

Testing

The project includes a comprehensive test suite to ensure reliability and code quality:

# Run all tests
./tests/run_tests.sh all

# Run specific test categories
./tests/run_tests.sh kernels          # Kernel operations
./tests/run_tests.sh update_r_libs    # R library management  
./tests/run_tests.sh config           # Configuration validation
./tests/run_tests.sh help             # Help and basic commands
./tests/run_tests.sh kernel_indexer   # Kernel indexing and collation

Test Features:

• Isolated Environments: Each test runs independently to prevent interference
• Mock Data: Tests use consistent mock kernel packages and catalogs
• Error Handling: Comprehensive testing of both success and failure scenarios
• Continuous Integration: Automated testing on all pull requests

Recent Improvements:

• Enhanced error handling with clear, descriptive error messages
• File path validation to prevent silent failures
• Improved test isolation for more reliable testing
• Better permission checking and validation
• Added kernel indexer test suite with 20+ tests

For detailed testing information, see the Contributing Guide.

Documentation

• User Guide: Detailed usage instructions
• Contributing Guide: Guidelines for contributors
• Kernel Indexer: Kernel indexing service documentation
• Web Interface: Web server documentation
• CHANGELOG: Project changelog