Feature Request: Support `.to()` Device Movement for Quantized Models

[EricRollei]

Feature Request: Support .to() Device Movement for Quantized Models

Summary

Enable moving quantized models (8-bit and 4-bit) between devices using .to(), allowing GPU→CPU→GPU movement without reloading from disk.

Motivation

Use Case: Fast Model Swapping in Multi-Model Workflows

When running inference pipelines that use multiple large models sequentially (e.g., image generation → upscaling → face restoration), users need to:

1. Run Model A on GPU
2. Free GPU VRAM for Model B
3. Run Model B on GPU
4. Restore Model A for next iteration

Current behavior:

• Calling .to('cpu') or .to('cuda') on a quantized model raises:

  ".to" is not supported for "8-bit" bitsandbytes models
  

• Users must delete the model and reload from disk (slow, 1-2+ minutes for large models)

Desired behavior:

• Move quantized model to CPU RAM (fast, ~10-20 seconds)
• Model stays in system RAM while other models use GPU
• Restore to GPU when needed (fast, ~10-20 seconds)

Real-World Example

Testing with Hunyuan Image 3.0 (13B parameters, ~81GB INT8):

• Reload from NVMe SSD: ~1:40 minutes
• Theoretical CPU→GPU transfer (if supported): ~10-20 seconds at DDR5 speeds

For iterative workflows generating 10+ images with model switching, this adds 15+ minutes of unnecessary I/O overhead.

Technical Context

Current Limitation

The quantization state (quant_state) appears to be tied to the original device. The Linear8bitLt and Linear4bit modules explicitly block .to() operations.

Suggested Approaches

1. Dequantize → Move → Requantize

   • Store original quantization config
   • Dequantize weights to full precision on source device
   • Move dequantized weights to target device
   • Requantize on target device
   • Higher memory peak during move, but enables the workflow

2. Direct State Movement

   • Move the underlying tensors and quant_state together
   • Update device references in quant_state
   • May require CUDA↔CPU implementations for quantization ops

3. Lazy Re-quantization

   • Move dequantized weights to CPU
   • Store quantization config
   • Re-quantize on-demand when .to('cuda') is called

Environment

• bitsandbytes version: 0.44.x / 0.45.x
• PyTorch: 2.0+
• Hardware: Multi-GPU and high-RAM systems where CPU parking is viable

Workaround Attempts

We attempted to:

1. Manually iterate through modules and move non-quantized components
2. Clear CUDA cache and use the model from CPU
3. Save/reload from a RAM disk

None provide the seamless experience of true .to() support.

Impact

This would benefit:

• ComfyUI/Automatic1111 users: Multi-model workflows are common
• Production inference pipelines: Model orchestration with limited VRAM
• Research workflows: Comparing quantized vs full-precision models

Related Issues

• [Link to any existing issues if found]

Proposed API

# Current (fails)
model = model.to('cpu')  # Raises error

# Desired
model = model.to('cpu')  # Moves model to CPU, preserves quantization state
model = model.to('cuda:0')  # Moves back, ready for inference

# Alternative explicit API
model = model.dequantize_and_move('cpu')  # If implicit .to() is too complex
model = model.move_and_requantize('cuda:0')

Willingness to Contribute

I'm happy to help test any proposed implementation or provide additional use case details.

---

Thank you for considering this feature request!

[matthewdouglas]

Hi,

Moving Linear8bitLt and Linear4bit should work in the latest bitsandbytes release. My first suggestion here would be to upgrade to bitsandbytes>=0.48.2 and see if that resolves your issue. Note that support for this is limited to PyTorch 2.3+.

Secondly, additional context could help here. It's not clear what model represents. Are you using transformers, diffusers, some other library? One of the libraries we integrate with may be responsible for producing the message about not being able to move the model; e.g. newer transformers versions might have started to allow model.to() for 4bit but not 8bit models. With that said, you should still be able to move Linear8bitLt and Linear4bit on your own as per the "Direct State Movement" indicated in suggested approaches.

If this does not solve your problem, please provide additional context such as a minimal reproducer and exact environment information (OS, python version, library versions for e.g. bitsandbytes, pytorch, accelerate, transformers, diffusers, any other relevant modeling library, etc).

[EricRollei]

Thank you for the helpful reply, much appreciated as it solved one of our two challenges. One remains however - please see below:

Follow-up: .to() Works in 0.48.2! + New Question About Low VRAM Loading

Great News: .to() Now Works! 🎉

After upgrading to bitsandbytes 0.48.2, the .to() functionality for NF4 quantized models now works correctly! We can successfully:

# Move NF4 model to CPU
model.to('cpu')  # ✓ Works!

# Move back to GPU  
model.to('cuda:0')  # ✓ Works!

This enables our "soft unload" workflow where we park the model in CPU RAM while other models use the GPU, then restore quickly (~30-40 seconds) instead of reloading from disk (~2 minutes).

Tested Environment:

• bitsandbytes 0.48.2
• PyTorch 2.7.1+cu128
• CUDA 12.8
• Model: Hunyuan Image 3.0 (13B params, ~46GB NF4 with attention in full precision)

Thank you to the team for implementing this!

---

New Challenge: Loading NF4 Models on 24-32GB GPUs

We now have a related but different challenge: enabling users with RTX 4090 (24GB) or RTX 5090 (32GB) to load and run NF4 quantized models.

The Problem

Our pre-quantized NF4 model:

• On-disk size: ~45GB
• VRAM required: ~46GB (because attention layers are kept in full precision for quality)
• Target GPUs: 24-32GB

The gap is 13-21GB that needs to live somewhere.

What We've Tried

1. Using device_map="auto" with max_memory

   max_memory = {0: "20GiB", "cpu": "100GiB"}
   model = AutoModelForCausalLM.from_pretrained(
       model_path,
       device_map="auto",
       max_memory=max_memory,
       low_cpu_mem_usage=True,
   )

   Result: Error - "Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model."

2. Setting llm_int8_enable_fp32_cpu_offload=True

   • Updated the saved model's config.json to include this flag
   • Result: Different error - RuntimeError: Tensor.item() cannot be called on meta tensors
   • This appears to be a bug in the interaction between bitsandbytes quantized layers and accelerate's CPU offload hooks

3. Custom device_map forcing NF4 layers to GPU

   • Works, but defeats the purpose since we can't fit 45GB on 24GB

Questions for the Team

1. Is CPU offload for NF4 layers fundamentally unsupported?

   • We understand NF4 compute happens on GPU, but can the weights live on CPU and be moved on-demand per forward pass?

2. Is there a recommended approach for models larger than VRAM?

   • Sequential layer loading?
   • Different quantization format?
   • Something we're missing?

3. The meta tensor error with llm_int8_enable_fp32_cpu_offload=True - is this a known issue?

   File "bitsandbytes/functional.py", line 525, in as_dict
       "nested_offset": self.offset.item(),
   RuntimeError: Tensor.item() cannot be called on meta tensors
   

Model Details

• Base model: Tencent Hunyuan Image 3.0 (13B parameters)
• Quantization: NF4 with double quantization
• Modules NOT quantized (kept in bfloat16):
  • VAE encoder/decoder
  • Attention projections (q, k, v, o)
  • Embeddings, layernorms
• Why attention in full precision: Quality degradation when quantized

What Would Help Our Users

Many users have RTX 4090 (24GB) or upcoming RTX 5090 (32GB). If there's any way to run a ~45GB NF4 model on these cards - even with CPU offload overhead - it would unlock this model for a huge user base.

We're happy to:

• Test any experimental features
• Provide detailed logs/profiles
• Create a minimal reproduction case

Thank you for your continued work on bitsandbytes!