← Back to blog
January 2026 Barrows 01

Training 70B Models on Consumer Hardware (If You're Willing to Wait)

What if the constraint "must load model to train it" was assumed rather than required?

Training a 70-billion parameter language model requires approximately 782 GB of memory when using the Adam optimizer. Weights, momentum, variance, gradients. The hardware to do this costs around $100,000: eight A100 GPUs at 80 GB each, high-speed interconnects, the whole enterprise setup.

Or you could do it with ~8 GB of system RAM, a modest GPU for compute, and patience.

The GPU VRAM holds the active layer during forward/backward passes. The system RAM holds the OS page cache for memory-mapped files. The SSD holds everything else. Total system: a gaming PC.

I've been working on a technique I'm calling frozen incremental training. Simple once you see it. I haven't found prior work that treats disk as the authoritative model state while preserving full-weight updates.

The Assumption Everyone Makes

Here's the standard gradient descent update, the formula at the heart of every neural network training loop:

W_new = W_old - learning_rate × gradient

Look at it carefully. To compute W_new, you need W_old loaded in memory. This is why training requires holding the entire model in RAM. Every framework, every paper, every implementation assumes this.

But watch what happens when we rewrite the same operation differently:

W += (-learning_rate × gradient)

Mathematically identical. Same result. Different implementation.

The insight: An additive accumulation to a memory-mapped file doesn't require holding the full model in RAM. At the OS level, mmap updates still fault pages into RAM via the page cache. The win is bounded residency, not zero reads. You only need enough RAM for the active layer's working set.

The weights live on disk. You compute the gradient for one layer. You add the update directly to the disk-backed storage. The full model is never resident in memory.

How It Works

The architecture:

+------------------------------------------------------------------+
|                       SYSTEM RAM (~8 GB)                         |
|                                                                  |
|   +-----------------+  +-----------------+  +-----------------+  |
|   | Layer N Weights |  |  Layer N Grads  |  |   Activations   |  |
|   |    (~3.5 GB)    |  |    (~3.5 GB)    |  |   (streamed)    |  |
|   +--------+--------+  +--------+--------+  +--------+--------+  |
|            |                    |                    |           |
|          load                compute              stream         |
|            |                    |                    |           |
+------------|--------------------|--------------------|------------+
             |                    |                    |
             |      +=delta       |                    |
             v                    v                    v
+------------------------------------------------------------------+
|                       NVMe SSD (~800 GB)                         |
|                                                                  |
|   +-----------------------------------------------------------+  |
|   | FROZEN STORAGE (memory-mapped files)                      |  |
|   |                                                           |  |
|   |   +---------+ +---------+ +---------+       +---------+   |  |
|   |   | Layer 0 | | Layer 1 | | Layer 2 |  ...  | Layer 79|   |  |
|   |   |  W,M,V  | |  W,M,V  | |  W,M,V  |       |  W,M,V  |   |  |
|   |   +---------+ +---------+ +---------+       +---------+   |  |
|   |                                                           |  |
|   |   + activation_cache.bin (temporary, streamed)            |  |
|   +-----------------------------------------------------------+  |
+------------------------------------------------------------------+

Training flow:
  1. Load Layer N weights from frozen storage into RAM
  2. Forward pass, stream activations to disk
  3. Backward pass, compute gradients
  4. frozen_W[:] += delta  (write-through to mmap)
  5. mmap.flush()
  6. Free Layer N from RAM, load Layer N+1

Storage

Three memory-mapped files per layer:

For a 70B model, this is ~782 GB of files. They sit on an SSD. Cold storage. Not RAM.

Training Loop

For each layer, sequentially:

  1. Load this layer's weights into RAM (~3.5 GB for a 70B model)
  2. Forward pass through this layer, stream activations to temporary disk storage
  3. Backward pass, compute gradients (reload activations from disk)
  4. Load M and V for this layer from disk
  5. Compute Adam update, tracking global timestep t
  6. Add delta directly to frozen W: frozen_W[:] += delta
  7. Flush M and V updates to disk
  8. Free this layer from RAM, move to next

Peak RAM: one layer's weights plus one layer's gradient plus activations. For a 70B model with ~80 layers, that's roughly 6-7 GB. Not 782 GB.

The Math

For the skeptical (as you should be), here's the Adam update rewritten for frozen storage:

# Traditional Adam (requires W, M, V in memory)
m = β₁ × m + (1 - β₁) × gradient
v = β₂ × v + (1 - β₂) × gradient²
m_hat = m / (1 - β₁^t)      # t = global timestep (scalar, incremented each batch)
v_hat = v / (1 - β₂^t)
W = W - lr × m_hat / (√v_hat + ε)

# Frozen incremental (on disk, updated via memmap)
frozen_m[:] = β₁ × frozen_m + (1 - β₁) × gradient
frozen_v[:] = β₂ × frozen_v + (1 - β₂) × gradient²
m_hat = frozen_m / (1 - β₁^t)  # t tracked globally, persisted to disk
v_hat = frozen_v / (1 - β₂^t)
delta = -lr × m_hat / (√v_hat + ε)
frozen_W[:] += delta  # Additive accumulation, write-through to file

Same update rule per parameter. Same convergence behavior, with bounded deviations in bias correction under sequential updates. The timestep t is a single scalar tracked globally and persisted alongside the model state. If training resumes from checkpoint, t picks up where it left off. (Note: if using progressive layer locking, per-layer timesteps would preserve exact Adam semantics. The global t approach means bias correction becomes approximate for locked layers.)

Verification

I ran three verification tests:

Test 1: Mathematical Equivalence

Traditional: W = [1.0, 2.0, 3.0] → [0.985, 1.970, 2.955]
Frozen:      W = [1.0, 2.0, 3.0] → [0.985, 1.970, 2.955]

✓ Results identical to floating-point precision

Test 2: Memory-Mapped Persistence

Write 1: arr[:] = [1.0, 2.0] → mmap.flush() → close
Write 2: arr[:] += [0.5, 0.3] → mmap.flush() → close
Read:    arr[:] = [1.5, 2.3]

✓ Values accumulate correctly across file handles

The explicit flush() ensures data integrity. The OS writes dirty pages to disk before we close the handle. Without it, you're at the mercy of the kernel's write-back timing.

Safety note: If the process crashes between the += and flush(), the model state can corrupt. Production implementations should checkpoint the global timestep and use epoch-level snapshots. For fine-tuning runs measured in hours, this is manageable. For week-long training, you want atomic writes or shadow copies.

Test 3: Training Convergence

Epoch 1: Loss 1.3381
Epoch 2: Loss 1.0878  ↓ 19%
Epoch 3: Loss 1.0259  ↓ 6%
Epoch 4: Loss 0.9866  ↓ 4%
Epoch 5: Loss 0.9573  ↓ 3%

✓ Loss decreases consistently

The model trains. Weights update. Loss goes down. The full model is never resident. (These tests were run on a small transformer under 100M params to validate mechanics, not throughput. 7B+ scale testing is ongoing.)

What About Existing Techniques?

There's a lot of work on memory-efficient training. Here's how this approach differs:

Earlier out-of-core and disk-offload approaches treat disk as overflow. This treats disk as the authoritative state and never materializes a full in-memory model.

Technique Approach Still Requires
Gradient Checkpointing Trade compute for memory during backprop Full model loaded
Model Parallelism Split model across GPUs Multiple GPUs
ZeRO (DeepSpeed) Partition optimizer states across GPUs Multiple GPUs
CPU Offloading Swap to CPU RAM during training Eventually loads full model
LoRA Train small adapter matrices only Not full model training
Frozen Incremental Additive accumulation to frozen storage Single GPU + SSD

The key difference: the full model is never resident in memory. Not partitioned across GPUs. Not swapped to CPU. The weights live on disk, and updates are accumulated directly to that frozen storage via the OS page cache as a sliding window.

ZeRO and CPU offloading still use the traditional W_new = W_old - lr × grad formulation. They just move tensors around to fit them in available memory. Frozen incremental training reformulates the operation itself.

Progressive Layer Locking

Once training happens layer-by-layer with frozen storage, you can lock layers that have converged.

Epoch 5: Loss 0.9573
→ layer_0 gradient magnitude below threshold
→ Lock layer_0 (skip gradient computation)

Epoch 6: Loss 0.9356  ↓ (layer_0 frozen, still converging)
Epoch 7: Loss 0.9296  ↓

Locked layers become immutable. No gradient computation, no updates, no I/O for that layer. Compute redirects to layers still learning.

The slow speed becomes a feature. Since we're processing layers individually anyway, we gain granular visibility into per-layer convergence. You can inspect, lock, and even rollback individual layers. Traditional batch training doesn't offer this introspection easily.

This is transfer learning without reloading the base model. The frozen layers stay on disk, untouched, while you fine-tune the upper layers with full weight updates (not adapter matrices).

The Numbers

Metric Traditional (70B) Frozen Incremental (70B)
Memory Required 782 GB (GPU/CPU) ~8 GB system RAM + SSD
Hardware 8× A100 (80GB) 1× consumer GPU + NVMe
Estimated Cost ~$100,000 ~$1,500
Time per Step ~seconds ~minutes
Memory Reduction 1× (baseline) ~100×

The disk requirement is the same (782 GB of model state). But disk is cheap. A 2TB NVMe SSD costs $150. The tradeoff: what takes hours on a cluster takes weeks or months on consumer hardware. For fine-tuning, that might be acceptable. For pretraining from scratch, probably not.

Why Hasn't Anyone Done This?

I've asked myself this question repeatedly. The math is trivial. Memory-mapped files are a 50-year-old operating system feature. Here's my working hypothesis:

Economic Incentives

Conceptual Barriers

The Obviousness Gap

It seems obvious in retrospect. Additive accumulation is mathematically equivalent to assignment. Memmap files exist in every OS. Layer-by-layer processing is standard. Variants of disk-offload training exist (ZeRO-Offload, etc.), but they generally treat disk as overflow rather than primary state.

The difference: treating the SSD as primary storage rather than overflow. Accepting the patience tradeoff as a feature rather than a bug.

The constraint was assumed, not derived. Once you question it, the solution is obvious.

Limitations

To be clear about what this approach doesn't solve:

The I/O Bottleneck (The Elephant)

This is the big one. Let's do the math honestly:

The ~7 GB/s assumes sequential access. If the mmap files fragment on disk, performance tanks to random I/O speeds (~100 MB/s). Pre-allocating contiguous files with fallocate before training is critical. Compare this to A100 HBM2 bandwidth (~2000 GB/s). Traditional training is orders of magnitude faster per step.

The tradeoff is space for time. This makes training possible on consumer hardware, not fast. For full pretraining on large datasets, you're looking at months or years. For fine-tuning (fewer epochs, smaller datasets), it becomes a viable "patience" strategy.

Other Considerations

The Broader Pattern

This fits into the same research program as Veriduct and OIS: questioning assumptions that security and ML systems take for granted.

Each challenges a foundational assumption by asking: is this constraint necessary, or just conventional?

The interpreter defines meaning. The bytes have none inherently.

For neural networks, the "interpreter" is the training loop. And the training loop doesn't actually require what we assumed it did.

What This Means

This doesn't make consumer hardware competitive with A100 clusters for speed. It makes training possible where it was previously impossible.

The practical applications are narrower than "train GPT-5 on your laptop":

It's not "training frontier models is now free." It's "training frontier models is now possible without institutional resources, if you're willing to wait."

That's a different world than the marketing pitch. But it's still a world that didn't exist before.

Chris Aziz, Bombadil Systems

For questions or collaboration: [email protected]