ggml: avoid creating CUDA context during device init

[ServeurpersoCom]

Make sure to read the contributing guidelines before submitting a PR

ggml_cuda_init() calls cudaSetDevice() on every GPU just to query free VRAM for logging. This triggers the creation of a CUDA primary context (120-550 MB depending on GPU), which is irreversible for the lifetime of the process. Every process that loads the backend pays this cost, even if it never uses the GPU (router mode).

This PR removes cudaSetDevice + cudaMemGetInfo from device init. The log loses the free VRAM part but still shows total VRAM via cudaGetDeviceProperties (no context needed). Free VRAM is queried later by FIT through its own cudaSetDevice path, so the context creation is simply deferred to first real use.

Tested on RTX PRO 6000 Blackwell. Router process not in nvidia-smi anymore. Model loading unchanged.

Fixes #20582

[am17an]

I guess this effectively reverts #20185?

[ServeurpersoCom]

I guess this effectively reverts #20185?

Not quite a git revert, but almost!
Total VRAM in the header: found 1 CUDA device (Total VRAM: 98304 MiB) -> keep
Total VRAM per device: VRAM: 98304 MiB -> keep

Free VRAM per device: (95000 MiB free) -> removed (causes cudaSetDevice)

[am17an]

cc @tehsiuhuang

[tehsiuhuang]

@ServeurpersoCom Thanks for finding this!! For learning purpose.. how did you measure the "creation of a CUDA primary context (120-550 MB depending on GPU)". Suprisingly this is really high.

[ServeurpersoCom]

@ServeurpersoCom Thanks for finding this!! For learning purpose.. how did you measure the "creation of a CUDA primary context (120-550 MB depending on GPU)". Suprisingly this is really high.

Completely at random, actually, since I have an RTX PRO 6000 with 96 GB of memory, and it takes less on a smaller GPU, I wanted to include "less than or equal to" so as not to say 550MB for everyone.

[ServeurpersoCom]

According to NVIDIA internal team (developer forums, 2018): "The pre-allocated memory amount is related to GPU SMs number. The GPU with more SMs requires a larger memory." There is no public API to predict or measure the context overhead separately.
The primary context created by cudaSetDevice contains: per-SM local memory stack pool (scales linearly with SM count, dominant factor), GPU page tables for virtual address translation (scales with total VRAM size), ELF/SASS module metadata (even in CUDA_MODULE_LOADING=LAZY mode which is the default since CUDA 11.7), and driver internal structures (stream descriptors, event pools, scheduling state).
This explains why the cost varies across GPUs: RTX 3060 with 28 SMs and 12 GB VRAM pays around 120 MB, dual RTX 5090 around 300 MB per GPU, and RTX PRO 6000 Blackwell with 192 SMs and 96 GB VRAM pays around 550 MB. The ratio roughly follows SM count.

[tehsiuhuang]

@ServeurpersoCom Thanks for finding this!! For learning purpose.. how did you measure the "creation of a CUDA primary context (120-550 MB depending on GPU)". Suprisingly this is really high.

Completely at random, actually, since I have an RTX PRO 6000 with 96 GB of memory, and it takes less on a smaller GPU, I wanted to include "less than or equal to" so as not to say 550MB for everyone.

Thanks!!! Just wrote a small test to understand the memory usage on A100/80GB.. I think your observation is accurate :)

Stage	API Called	Per-GPU Memory	Delta
0	(none)	4 MiB	—
1	cudaGetDeviceCount + cudaGetDeviceProperties	4 MiB	+0 MiB
2	cudaSetDevice	423 MiB	+419 MiB
3	cudaMemGetInfo	423 MiB	+0 MiB