Automation for GPU layers, tensor split, tensor overrides, and context size (with MoE optimizations)

[JohannesGaessler]

CPU + GPU hybrid inference has been a core feature of llama.cpp since early on, and I would argue, one of the major selling points vs. projects like ExLlama.
The way to control memory use until now was to manually set parameter like --n-gpu-layers and --tensor-split to fit memory use to free VRAM.
However, this is of course suboptimal in terms of usability.
Downstream projects like Ollama and KoboldCpp have implemented mechanisms for automating memory allocation but those rely on rough heuristics and tend to be inaccurate.
As a consequence, to avoid running out of memory in some cases the heuristics are rather conservative and leave potential performance on the table.
The problem becomes even harder when running models across multiple GPUs or when running MoE models where the dense tensors
should be prioritized over the sparse MoE tensors for optimal performance.

On the latest llama.cpp version following #16653 I implemented code to automate memory allocations across GPUs.
It works by doing virtual test allocations and using those as feedback to iteratively reduce memory use until the model fits across all GPUs.
The metric for memory use is the same as in the "memory breakdown" that you may have seen in recent llama.cpp versions.
The implementation is generic and should work for any ggml backend as long as it supports CPU + GPU hybrid inference and the memory breakdown is correct.
If you encounter problems using this new functionality, please open an issue instead of commenting here as this will make the process easier from my side.

The code starts by first checking whether the model is projected to fit as-is.
If yes, no changes are made.
If not, it first reduces the context size to free up memory.
If that is still not enough it starts moving tensors from VRAM to RAM.
Dense tensors are prioritized for better MoE performance.
Ideally one would only assign whole layers to GPUs for simplicity.
However, as individual layers can be very large against "small" GPUs with only 24 GiB VRAM this would result in significant waste.
For this reason, layers can "overflow", meaning that parts of them are moved to the next GPU in line or to system RAM.

Command-Line Interface

The fitting of runtime parameters can be controlled as follows:

• --fit, -fit: set to on by default, can be set to off to disable parameter fitting.
• --fit-target, -fitt: target amount of free memory to leave on each GPU. As of right now this is the same value for all GPUs and it is not possible to specify e.g. an amount that should be used regardless of free memory.
• --fit-ctx, -fitc: minimum context size that can be set automatically. If --ctx-size is explicitly set by the user it is not changed.
• If arguments like --n-gpu-layers, --tensor-split, or --override-tensor that affect memory allocation are set by the user, there is no change to that memory allocation. There is no support for automatic modification of only one of these arguments, they are either wholly under user control or wholly under program control.

There is a new tool llama-fit-params that can be used to retrieve the parameters that would be set by the new parameter fitting logic.
For example:

> $ ./build/bin/llama-fit-params --model models/opt/gpt-oss-120b-mxfp4-00001-of-00003.gguf -ub 4096 -b 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
  Device 1: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
build: 7413 (ae534ec0c) with GNU 15.2.1 for Linux x86_64
llama_params_fit_impl: projected memory use with initial parameters [MiB]:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 4090):  24080 total,  34873 used,  11187 deficit
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 4090):  24080 total,  31847 used,   8161 deficit
llama_params_fit_impl: projected to use 66721 MiB of device memory vs. 48161 MiB of free device memory
llama_params_fit_impl: cannot fulfill margin of 1024 MiB on all devices, need to use 21397 MiB less in total
llama_params_fit_impl: context size reduced from 131072 to 4096 -> need 4490 MiB less memory in total
llama_params_fit_impl: with only dense weights in device memory there is a total surplus of 42064 MiB
llama_params_fit_impl: filling dense-only layers back-to-front:
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 4090): 36 layers,   2201 MiB used,  21484 MiB free
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 4090):  0 layers,    985 MiB used,  22700 MiB free
llama_params_fit_impl: converting dense-only layers to full layers and filling them front-to-back with overflow to next device/system memory:
llama_params_fit_impl:   - CUDA0 (NVIDIA GeForce RTX 4090): 14 layers ( 1 overflowing),  22576 MiB used,   1109 MiB free
llama_params_fit_impl:   - CUDA1 (NVIDIA GeForce RTX 4090): 22 layers (11 overflowing),  22208 MiB used,   1477 MiB free
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 8.81 seconds
Printing fitted CLI arguments to stdout...
-c 4096 -ngl 37 -ts 14,23 -ot blk\.13\.ffn_(up|gate|down).*=CUDA1,blk\.25\.ffn_down.*=CPU,blk\.26\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.27\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.28\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.29\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.30\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.31\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.32\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.33\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.34\.ffn_(up|down|gate)_(ch|)exps=CPU,blk\.35\.ffn_(up|down|gate)_(ch|)exps=CPU

Benchmark

As of right now llama-bench does not have support for -fit, -fitt, and -fitc.
For this reason, the following workaround was used to feed the results from llama-fit-params into llama-bench:

./build/bin/llama-fit-params -m models/opt/${model_name}-${quantization}.gguf -b 4096 -ub 4096 | tee tmp.txt
./build/bin/llama-bench -m models/opt/${model_name}-${quantization}.gguf -r 1 -fa 1 $(tail -c +17 tmp.txt | tr ',' ';')

The benchmark was done on a system with an AMD EPYC 7742 CPU and 8 3200 "MHz" DIMMs.

Model	GPUs	Time to fit [s]	Fully in VRAM?	VRAM utilization	pp4096 [t/s]	tg128 [t/s]
Qwen 3 Next BF16	None	-	No	-	38.89	6.23
Qwen 3 Next BF16	1x RTX 4090	4.89	No	88.1%	381.52	19.01
Qwen 3 Next BF16	2x RTX 4090	7.75	No	88.5%	246.29	20.89
Qwen 3 Next BF16	3x RTX 4090	10.70	No	88.3%	340.88	22.00
Qwen 3 Next BF16	4x RTX 4090	13.87	No	89.3%	433.10	24.70
Qwen 3 Next BF16	4x RTX 4090, 1x RTX 5090	16.93	No	89.7%	526.71	26.19
Qwen 3 Next BF16	4x RTX 4090, 1x RTX 5090, 1x RTX 3090	20.39	No	90.2%	599.86	31.37
Qwen 3 Next q8_0	None	-	No	-	44.81	7.17
Qwen 3 Next q8_0	1x RTX 4090	4.98	No	87.3%	904.49	24.26
Qwen 3 Next q8_0	2x RTX 4090	7.51	No	88.5%	574.43	28.34
Qwen 3 Next q8_0	3x RTX 4090	10.22	No	89.3%	1086.23	33.33
Qwen 3 Next q8_0	4x RTX 4090	12.19	Yes	87.0%	2474.67	41.37
GPT OSS 120b mxfp4	None	-	No	-	115.78	23.63
GPT OSS 120b mxfp4	1x RTX 4090	5.56	No	83.7%	1733.20	52.09
GPT OSS 120b mxfp4	2x RTX 4090	10.48	No	89.4%	2452.52	78.27
GPT OSS 120b mxfp4	3x RTX 4090	11.47	Yes	86.0%	5499.52	180.29
GPT OSS 120b mxfp4	4x RTX 4090	1.55	Yes	68.2%	5219.51	182.89

The VRAM utilization is at ~85-90%.
As the default --fit-target is 1024 MiB, that would ideally leave ~4% of free VRAM on each GPU.
However, since individual tensors can be several GB in size some amount of waste is inevitable.

The time to fit the parameters increases roughly linearly with the number of GPUs.
Under ideal circumstances such as when running GPT OSS 120b on 4x RTX 4090 the code only needs to check that the VRAM is sufficient.
For Qwen 3 Next there currently seems to be a bug where the memory needed for the context is not accounted correctly so a full fit is done.
Time to fit is still fairly unoptimized.

Performance mostly increases as VRAM use increases, except when going from a single GPU to two GPUs
(while still being bottlenecked by RAM) or when the model could already be fit on fewer GPUs.
With better multi GPU code the performance should increase monotonically as more GPUs are added.