[Ecosystem] Kubetorch

[py-rh]

Contact emails

paul@run.house, donny@run.house

Project summary

A Fast, Pythonic Interface for Running ML Workloads on Kubernetes

Project description

Kubernetes has emerged as the default compute foundation for ML (outside of pre-training on HPC-style Slurm clusters). However, the development experience is terrible. Making a minor code change to your distributed training workload might lead to 10-30 minutes before the first training batch. And the smallest OOM or node/pod pre-emption causes your entire workload to fall over, with no recourse for faults.

With Kubetorch, the first benefit is extremely fast iteration on your distributed Torch trainings; all compute, artifacts, and state is retained during iteration, allowing for iteration on distributed training to occur in <2 seconds. Secondly, it comes with Pythonic APIs that are friendly and familiar to practitioners, with no need to delve into YAML or Dockerfiles too early. But it builds on purely Kubernetes-native primitives, so it is easy to manage as part of the regular software platform and plugs in to the rest of the cloud-native ecosystem. Finally, it offers great programmatic fault tolerance, as the driver / controller Python making the call lives outside of the process group doing the training; if you hit a CUDA OOM or lose a pod to pre-emption, control is simply returned to the driver process and it can make a decision on how to continue (reduce batch size / reform process group with smaller world size, respectively).

Kubetorch is partially named as an analogy to how Torch makes GPUs usable in Python with the .to() APIs -- in the same way, Kubetorch uses .to() to take regular Python classes (a Trainer) and functions (your training entrypoint) and launch them on Kubernetes while returning a callable that can be called locally (from within CI, production orchestration).

A brief code snippet showing the APIs to launch a distributed training

def train(epochs, batch_size=32):
    torch.distributed.init_process_group(backend="nccl")
    ....
    model = DDP(model, device_ids=[device_id])
    for epoch in range(epochs):
        ...

if __name__ == "__main__":
    gpus = kt.Compute(
        gpus=8,
        image=kt.Image(image_id="nvcr.io/nvidia/pytorch:23.10-py3"),
        launch_timeout=600,
    ).distribute("pytorch", workers=4)

    train_ddp = kt.fn(train).to(gpus)

    results = train_ddp(epochs=args.epochs, batch_size=args.batch_size)

Are there any other projects in the PyTorch Ecosystem similar to yours? If, yes, what are they?

The closest analogue is Ray Core; it would be acceptable to think of Kubetorch as Kubernetes-native actor deployments, where the controller is actually the driver process doing launching and making calls, and the actors are 1 or more pods.

Project repo URL

https://github.com/run-house/kubetorch

Additional repos in scope of the application

https://github.com/run-house/kubetorch-examples

Project license

Apache 2.0

GitHub handles of the project maintainer(s)

@CarolineChen @jlewitt1 @dongreenberg @BelSasha @mkandler @py-rh

Is there a corporate or academic entity backing this project? If so, please provide the name and URL of the entity.

Runhouse run.house

Website URL

run.house

Documentation

https://www.run.house/kubetorch/introduction
https://www.youtube.com/@runhouse_

How do you build and test the project today (continuous integration)? Please describe.

Tests running in GitHub
https://github.com/run-house/kubetorch/tree/main/python_client/tests

Version of PyTorch

PyTorch 1.0+ all work, the library is not opinionated

Components of PyTorch

We don't use PyTorch as a dependency, but our primary goal is to launch and run PyTorch-based workloads.

How long do you expect to maintain the project?

Forever

Additional information

No response

[lantiga]

Thank you for the great contribution to the ecosystem! Things are in good shape, once the points below are addressed we can quickly proceed to a vote.

Functional Requirements:

✅ 1. PyTorch Support: Project must demonstrate benefit to the pytorch community and be validated with the latest versions of PyTorch.
✅ 2. Working CI: Projects must have a demonstrable working CI workflow for all declared architectures
🟡 3. Governance: Projects must have documented technical governance defined and implemented and it must be represented in their Github or Gitlab repository. Example: Review the LF’s Minimum Viable Governance framework.
The project is built by Runhouse, but more visibility on governance would be ideal, e.g. are there core maintainers, are they all at Runhouse, are there codeowners
✅ 4. User experience: The value proposition should be made clear for the project and documented in the README of the project repository.
✅ 5. Quick start: Project should have a working and demonstrable quick start.
✅ 6. Documentation: Projects must have clear and comprehensive documentation.
✅ 7. Permissive licensing: Users must be able to utilize ecosystem projects without licensing concerns. e.g. BSD-3, Apache-2 and MIT licenses
✅ 8. Functional Testing: Users should have developed tests submitted as a part of their project, and be confident that the project will work well with the latest releases of PyTorch.
✅ 9. Packages: Projects need to have support for binary installation options (pip/Conda).
✅ 10. Ongoing maintenance: Project maintainers must be committed to supporting and maintaining their projects for the next 12 months at a minimum. Projects must commit to updating their projects to the latest two releases of PyTorch (e.g. versions 2.6, 2.5.1).

Measurable Requirements:

✅ 1. Core Maintainers: The project must have a minimum of 2 core maintainers
✅ 2. Contributors: The project should have a minimum of 5 total contributors active in the last 90 days.
✅ 3. Commits: The project must have 30 commits in the last 90 days. This is off by like 15 days: it shouldn't be a deal breaker.
✅ 4. Stars: The project must have a minimum of 200 stars on github

Github Layout Requirements:

✅ 1. LICENSE.md at the root of the repository specifying the terms and conditions for using, distributing, and modifying the software. In addition, you should provide information on the license of any third-party code included in the project.
✅ 2. README.md welcomes new community members to the project and explains why the project is useful and how to get started. Should include information about release methodology, cadence, and criteria in the README.
✅ 3. CONTRIBUTING.md explains how to contribute to the project. The file explains the types of contributions needed and how the development process works.
⛔️ 4. CODEOWNERS that define individuals or teams responsible for code in a repository; document current project owners and Retired committers.
⛔️ 5. CODE_OF_CONDUCT.md sets the ground rules for participants’ behavior and helps facilitate a friendly, welcoming environment. By default, projects should leverage the [Linux]([https://lfprojects.org/policies/code-of-conduct/) [Foundation](https://lfprojects.org/policies/code-of-conduct/) [Code](https://lfprojects.org/policies/code-of-conduct/) of [Conduct](https://lfprojects.org/policies/code-of-conduct/) unless](https://lfprojects.org/policies/code-of-conduct/)%C2%A0%5B%5BFoundation%5D(https://lfprojects.org/policies/code-of-conduct/)%5D(https://lfprojects.org/policies/code-of-conduct/)%C2%A0%5B%5BCode%5D(https://lfprojects.org/policies/code-of-conduct/)%5D(https://lfprojects.org/policies/code-%5Bof%5D(https://lfprojects.org/policies/code-of-conduct/)-conduct/)%C2%A0%5Bof%5D(https://lfprojects.org/policies/code-of-conduct/)%C2%A0%5B%5BConduct%5D(https://lfprojects.org/policies/code-of-conduct/)%5D(https://lfprojects.org/policies/code-of-conduct/)%C2%A0unless) an alternate Code of Conduct is approved prior.

[py-rh]

Thanks and happy New Year @lantiga - updated with conduct + contributing (w/ code owners).