Added CUDA support by zpitroda · Pull Request #228 · mindverse/Second-Me

zpitroda · 2025-04-14T18:58:39Z

Added:

CUDA support through build option
Memory management including layer offloading
Ollama model release after training
Switch to toggle CPU/GPU training

Current issues:

After training finishes and service starts, model takes a few minutes to load during which the chat will return empty responses
No way to toggle between CPU/GPU during inference
Only tested in docker with wsl backend on windows
I believe current commit still has bug causing training to always use 0.5B model

To do:

Optimize training by removing unnecessary functions and parameters
Update/improve documentation

Technical changes:

Created new docker file Dockerfile.backend.cuda to handle building with cuda support including using a docker image prebuilt with nvidia drivers, buildtools, and python 3.12
Added scripts prompt_cuda.sh and prompt_cuda.bat that ask the user if they want to build with CUDA support when running make docker-up.
The prompt scripts now generate a docker-compose.override.yml file to explicitly tell Docker Compose which Dockerfile (Dockerfile.backend or Dockerfile.backend.cuda) to use based on the user's choice.
The backend API routes_l2.py includes an endpoint /api/kernel2/cuda/available that uses torch.cuda.is_available() to check if CUDA is usable at runtime.
The training interface TrainingConfiguration.tsx calls the backend API to check CUDA availability and enables/disables the "Enable CUDA GPU Acceleration" toggle accordingly.

yingapple · 2025-04-15T02:07:40Z

i'm sorry, but there are some conflicts.

yingapple · 2025-04-15T02:09:45Z

Thank you especially for your outstanding contribution. I’m really looking forward to merging this PR.

zpitroda · 2025-04-15T02:11:58Z

I'll try to have those conflicts worked out asap 🫡

zpitroda · 2025-04-15T15:00:30Z

Merge conflicts should be worked out!

- CUDA detection - Memory handling - Ollama model release after training

added cuda support flag so log accurately reflected cuda toggle

Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run

Enabled memory pinning and optimizer state offload

rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support

Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support

yingapple · 2025-04-16T11:44:46Z

Let’s ignore these new conflicts for now. I tested your branch, and it seems there’s a bit of an issue.
Below is the log after i chose the gpu mode.

[+] Building 2/2
✔ backend Built 0.0s
✔ frontend Built 0.0s
docker compose up -d
[+] Running 3/4
✔ Network second-me_second-me-network Created 0.0s
✔ Volume "second-me_llama-cpp-build" Created 0.0s
⠏ Container second-me-backend Starting 6.9s
✔ Container second-me-frontend Created 0.0s
Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

yingapple · 2025-04-16T11:51:35Z

I probably haven’t installed the GPU driver for Docker. Maybe this PR should also mention that in the README.

zpitroda · 2025-04-16T16:15:24Z

Yeah sorry I should've added that, it needs the Nvidia container toolkit installed, can I ask if you're running this on windows or Linux, and if Windows are you using the WSL 2 backend?

zpitroda · 2025-04-16T18:24:16Z

@yingapple I tried replicating your issue by completely resetting my docker and wsl environments but it seems to have all started fine. Please try installing the nvidia container toolkit with:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

then to test your gpu is detected

sudo docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi

Please let me know if this works for you and your system details. Thank you!

yingapple · 2025-04-17T06:33:51Z

@yingapple I tried replicating your issue by completely resetting my docker and wsl environments but it seems to have all started fine. Please try installing the nvidia container toolkit with:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

then to test your gpu is detected

sudo docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi

Please let me know if this works for you and your system details. Thank you!

I have run success on my a100 server! We are testing it on none gpu machine

yingapple · 2025-04-17T07:14:14Z

One Error is we should add "\n" before you modify the .env, otherwise the docker start fail

yingapple · 2025-04-17T07:32:33Z

@zpitroda Additionally:
I suggest having a separate docker-compose.gpu.yml.
Otherwise, machines without a GPU will throw errors.

Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification

…cond-Me into feature/cuda-support

Last push accidentally broke cuda toggle

zpitroda · 2025-04-17T19:56:40Z

@yingapple thanks for the suggestions! I've incorporated both of them and synced the branch with the main one!

Use cuda by default when using docker-restart-backend-fast

zpitroda · 2025-04-23T02:40:25Z

Noticed some bugs not sure if they're related to the recent push but working on sorting them out

nvm was fixed with last merge

Removed unnecessary makefile command and fixed gpu logging

yingapple · 2025-04-23T03:32:22Z

Noticed some bugs not sure if they're related to the recent push but working on sorting them out

nvm was fixed with last merge

I'm testing.

zpitroda · 2025-04-23T03:36:35Z

Noticed some bugs not sure if they're related to the recent push but working on sorting them out

nvm was fixed with last merge

I'm testing.

Yeah you beat me to it I was about to push a fix as well lol

3050226203 · 2025-04-23T08:32:07Z

@yingapple @3050226203 I believe I've implemented these fixes, please let me know how it looks and if there's anything else that should be changed!

The problem has been solved. I've found the model location. Thank you.

- Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments

Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu

zpitroda · 2025-04-24T03:18:09Z

Working on figuring out the cause but training is currently failing with a free(): double free detected in tcache 2 error

3050226203 · 2025-04-24T04:26:48Z

When calling local Ollama, the following problems will occur:
TypeError: Exception() takes no keyword arguments
2025-04-24 03:19:24 [ERROR] document_service.py:493 - Error processing document embedding: Exception() takes no keyword arguments
2025-04-24 03:19:24 [ERROR] trainprocess_service.py:128 - Generate document embeddings failed: Exception() takes no keyword arguments
2025-04-24 03:19:24 [ERROR] trainprocess_service.py:1089 - Step generate_document_embeddings failed
2025-04-24 03:19:24 [INFO] trainprocess_service.py:1090 - Marking step as failed: stage=generate_document_embeddings, step=generate_document_embeddings

…port

Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones

zpitroda · 2025-04-24T15:16:12Z

When calling local Ollama, the following problems will occur: TypeError: Exception() takes no keyword arguments 2025-04-24 03:19:24 [ERROR] document_service.py:493 - Error processing document embedding: Exception() takes no keyword arguments 2025-04-24 03:19:24 [ERROR] trainprocess_service.py:128 - Generate document embeddings failed: Exception() takes no keyword arguments 2025-04-24 03:19:24 [ERROR] trainprocess_service.py:1089 - Step generate_document_embeddings failed 2025-04-24 03:19:24 [INFO] trainprocess_service.py:1090 - Marking step as failed: stage=generate_document_embeddings, step=generate_document_embeddings

@3050226203 Please let me know if it works after my last push!

Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly

yingapple · 2025-04-25T02:17:49Z

I finished my test. Let's merge. And do regression testing. Your contribution will be in our first release version!

zpitroda · 2025-04-25T02:29:48Z

I finished my test. Let's merge. And do regression testing. Your contribution will be in our first release version!

Love to hear it! Last thing I'm just working out rn is making sure large models like the 7B work regardless of available memory I'll hopefully be able to push that tomorrow

* Add CUDA support - CUDA detection - Memory handling - Ollama model release after training * Fix logging issue added cuda support flag so log accurately reflected cuda toggle * Update llama.cpp rebuild Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run * Improved vram management Enabled memory pinning and optimizer state offload * Fix CUDA check rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support * Added fast restart and fixed CUDA check command Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support * Added docker-compose.gpu.yml Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification * Fixed cuda toggle Last push accidentally broke cuda toggle * Code review fixes Fixed errors resulting from removed code: - Added return save_path to end of save_hf_model function - Rolled back download_file_with_progress function * Update Makefile Use cuda by default when using docker-restart-backend-fast * Minor cleanup Removed unnecessary makefile command and fixed gpu logging * Delete .gpu_selected * Simplified cuda training code - Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments * Fixed gpu/cpu selection Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu * Fix Ollama embedding error Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones * Fixed model selection & memory error Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly

* feature: use uv to setup python environment * TrainProcessService add singleten method: get_instance * feat: fix code * Added CUDA support (#228) * Add CUDA support - CUDA detection - Memory handling - Ollama model release after training * Fix logging issue added cuda support flag so log accurately reflected cuda toggle * Update llama.cpp rebuild Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run * Improved vram management Enabled memory pinning and optimizer state offload * Fix CUDA check rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support * Added fast restart and fixed CUDA check command Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support * Added docker-compose.gpu.yml Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification * Fixed cuda toggle Last push accidentally broke cuda toggle * Code review fixes Fixed errors resulting from removed code: - Added return save_path to end of save_hf_model function - Rolled back download_file_with_progress function * Update Makefile Use cuda by default when using docker-restart-backend-fast * Minor cleanup Removed unnecessary makefile command and fixed gpu logging * Delete .gpu_selected * Simplified cuda training code - Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments * Fixed gpu/cpu selection Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu * Fix Ollama embedding error Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones * Fixed model selection & memory error Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly * fix: train service singlten --------- Co-authored-by: Zachary Pitroda <30330004+zpitroda@users.noreply.github.com>

* Add CUDA support - CUDA detection - Memory handling - Ollama model release after training * Fix logging issue added cuda support flag so log accurately reflected cuda toggle * Update llama.cpp rebuild Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run * Improved vram management Enabled memory pinning and optimizer state offload * Fix CUDA check rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support * Added fast restart and fixed CUDA check command Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support * Added docker-compose.gpu.yml Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification * Fixed cuda toggle Last push accidentally broke cuda toggle * Code review fixes Fixed errors resulting from removed code: - Added return save_path to end of save_hf_model function - Rolled back download_file_with_progress function * Update Makefile Use cuda by default when using docker-restart-backend-fast * Minor cleanup Removed unnecessary makefile command and fixed gpu logging * Delete .gpu_selected * Simplified cuda training code - Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments * Fixed gpu/cpu selection Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu * Fix Ollama embedding error Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones * Fixed model selection & memory error Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly

* feature: use uv to setup python environment * TrainProcessService add singleten method: get_instance * feat: fix code * Added CUDA support (mindverse#228) * Add CUDA support - CUDA detection - Memory handling - Ollama model release after training * Fix logging issue added cuda support flag so log accurately reflected cuda toggle * Update llama.cpp rebuild Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run * Improved vram management Enabled memory pinning and optimizer state offload * Fix CUDA check rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support * Added fast restart and fixed CUDA check command Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support * Added docker-compose.gpu.yml Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification * Fixed cuda toggle Last push accidentally broke cuda toggle * Code review fixes Fixed errors resulting from removed code: - Added return save_path to end of save_hf_model function - Rolled back download_file_with_progress function * Update Makefile Use cuda by default when using docker-restart-backend-fast * Minor cleanup Removed unnecessary makefile command and fixed gpu logging * Delete .gpu_selected * Simplified cuda training code - Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments * Fixed gpu/cpu selection Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu * Fix Ollama embedding error Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones * Fixed model selection & memory error Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly * fix: train service singlten --------- Co-authored-by: Zachary Pitroda <30330004+zpitroda@users.noreply.github.com>

* Add CUDA support - CUDA detection - Memory handling - Ollama model release after training * Fix logging issue added cuda support flag so log accurately reflected cuda toggle * Update llama.cpp rebuild Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run * Improved vram management Enabled memory pinning and optimizer state offload * Fix CUDA check rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support * Added fast restart and fixed CUDA check command Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support * Added docker-compose.gpu.yml Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification * Fixed cuda toggle Last push accidentally broke cuda toggle * Code review fixes Fixed errors resulting from removed code: - Added return save_path to end of save_hf_model function - Rolled back download_file_with_progress function * Update Makefile Use cuda by default when using docker-restart-backend-fast * Minor cleanup Removed unnecessary makefile command and fixed gpu logging * Delete .gpu_selected * Simplified cuda training code - Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments * Fixed gpu/cpu selection Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu * Fix Ollama embedding error Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones * Fixed model selection & memory error Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly

* feature: use uv to setup python environment * TrainProcessService add singleten method: get_instance * feat: fix code * Added CUDA support (mindverse#228) * Add CUDA support - CUDA detection - Memory handling - Ollama model release after training * Fix logging issue added cuda support flag so log accurately reflected cuda toggle * Update llama.cpp rebuild Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run * Improved vram management Enabled memory pinning and optimizer state offload * Fix CUDA check rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support * Added fast restart and fixed CUDA check command Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support * Added docker-compose.gpu.yml Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification * Fixed cuda toggle Last push accidentally broke cuda toggle * Code review fixes Fixed errors resulting from removed code: - Added return save_path to end of save_hf_model function - Rolled back download_file_with_progress function * Update Makefile Use cuda by default when using docker-restart-backend-fast * Minor cleanup Removed unnecessary makefile command and fixed gpu logging * Delete .gpu_selected * Simplified cuda training code - Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments * Fixed gpu/cpu selection Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu * Fix Ollama embedding error Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones * Fixed model selection & memory error Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly * fix: train service singlten --------- Co-authored-by: Zachary Pitroda <30330004+zpitroda@users.noreply.github.com>

zpitroda marked this pull request as draft April 15, 2025 23:56

zpitroda marked this pull request as ready for review April 16, 2025 04:51

zpitroda changed the title ~~Add CUDA support~~ Updated llama.cpp rebuild process Apr 16, 2025

zpitroda added 6 commits April 16, 2025 01:32

Add CUDA support

273e549

- CUDA detection - Memory handling - Ollama model release after training

Fix logging issue

91d5cc3

added cuda support flag so log accurately reflected cuda toggle

Update llama.cpp rebuild

c648cc3

Changed llama.cpp to only check if cuda support is enabled and if so rebuild during the first build rather than each run

Improved vram management

3bf6d0c

Enabled memory pinning and optimizer state offload

Fix CUDA check

3f86cfa

rewrote llama.cpp rebuild logic, added manual y/n toggle if user wants to enable cuda support

Added fast restart and fixed CUDA check command

ac10939

Added make docker-restart-backend-fast to restart the backend and reflect code changes without causing a full llama.cpp rebuild Fixed make docker-check-cuda command to correctly reflect cuda support

zpitroda force-pushed the feature/cuda-support branch from 05c10aa to ac10939 Compare April 16, 2025 05:33

yingapple mentioned this pull request Apr 17, 2025

Unable to enable GPU training #251

Open

Added docker-compose.gpu.yml

0489efe

Added docker-compose.gpu.yml to fix error on machines without nvidia gpu and made sure "\n" is added before .env modification

zpitroda closed this Apr 17, 2025

zpitroda force-pushed the feature/cuda-support branch from 0489efe to e1ae6f5 Compare April 17, 2025 18:23

Merge branch 'feature/cuda-support' of https://github.com/zpitroda/Se…

31a31c2

…cond-Me into feature/cuda-support

zpitroda reopened this Apr 17, 2025

Fixed cuda toggle

8839cb9

Last push accidentally broke cuda toggle

zpitroda changed the title ~~Updated llama.cpp rebuild process~~ Added CUDA support Apr 20, 2025

zpitroda added 3 commits April 22, 2025 13:52

Update Makefile

173dd2e

Use cuda by default when using docker-restart-backend-fast

Merge remote-tracking branch 'upstream/master' into feature/cuda-support

e1e16b4

Merge branch 'mindverse:master' into feature/cuda-support

cb7df57

Minor cleanup

45b5653

Removed unnecessary makefile command and fixed gpu logging

zpitroda changed the base branch from master to develop April 23, 2025 16:37

zpitroda added 4 commits April 23, 2025 14:05

Merge branch 'mindverse:master' into feature/cuda-support

6620b5b

Delete .gpu_selected

c9f9e63

Simplified cuda training code

b65cbf4

- Removed dtype setting to let torch automatically handle it - Removed vram logging - Removed Unnecessary/old comments

Fixed gpu/cpu selection

7a67e0d

Made "make docker-use-gpu/cpu" command work with .gpu_selected flag and changed "make docker-restart-backend-fast" command to respect flag instead of always using gpu

zpitroda requested a review from yingapple April 23, 2025 19:25

zpitroda added 2 commits April 24, 2025 10:58

Merge remote-tracking branch 'upstream/develop' into feature/cuda-sup…

ec2cd34

…port

Fix Ollama embedding error

2f5cd33

Added custom exception class for Ollama embeddings, which seemed to be returning keyword arguments while the Python exception class only accepts positional ones

Fixed model selection & memory error

7d1e748

Fixed training defaulting to 0.5B model regardless of selection and fixed "free(): double free detected in tcache 2" error caused by cuda flag being passed incorrectly

yingapple merged commit 0530909 into mindverse:develop Apr 25, 2025
1 check passed

yashuatla mentioned this pull request Jun 23, 2025

Feature implementation from commits 8ace28a..19adcac yashuatla/Second-Me#5

Open

Conversation

zpitroda commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yingapple commented Apr 15, 2025

Uh oh!

yingapple commented Apr 15, 2025

Uh oh!

zpitroda commented Apr 15, 2025

Uh oh!

zpitroda commented Apr 15, 2025

Uh oh!

yingapple commented Apr 16, 2025

Uh oh!

yingapple commented Apr 16, 2025

Uh oh!

zpitroda commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zpitroda commented Apr 16, 2025

Uh oh!

yingapple commented Apr 17, 2025

Uh oh!

yingapple commented Apr 17, 2025

Uh oh!

yingapple commented Apr 17, 2025

Uh oh!

zpitroda commented Apr 17, 2025

Uh oh!

zpitroda commented Apr 23, 2025

Uh oh!

yingapple commented Apr 23, 2025

Uh oh!

zpitroda commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

3050226203 commented Apr 23, 2025

Uh oh!

zpitroda commented Apr 24, 2025

Uh oh!

3050226203 commented Apr 24, 2025

Uh oh!

zpitroda commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yingapple commented Apr 25, 2025

Uh oh!

Uh oh!

zpitroda commented Apr 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zpitroda commented Apr 14, 2025 •

edited

Loading

zpitroda commented Apr 16, 2025 •

edited

Loading

zpitroda commented Apr 23, 2025 •

edited

Loading

zpitroda commented Apr 24, 2025 •

edited

Loading