| Training compute & data |
Larger training budgets, better data mixtures, curricula and
longer training can put more capability into a given model
size.22
|
Training-optimal and inference-optimal size differ. Synthetic
data quality, data coverage and teacher access can constrain
transfer.
|
Net shrink rate; no training hardware on phone |
| Distillation & post-training |
Teacher outputs, reasoning traces and reinforcement learning
can improve smaller models. DeepSeek released distilled
variants from 1.5B to 70B.23
|
Selected math/coding gains do not prove broad knowledge,
robustness or long-task equivalence. Teacher-generated tokens
still cost resources.
|
Net shrink rate; quality gate unresolved |
| Quantization & low-bit training |
4-bit packing gives a raw 4× weight reduction versus 16-bit.
Quantization-aware or native low-bit training can go
further.12, 13
|
Metadata, outliers and higher-precision operations reduce
savings. Quality can fall. Precision cannot halve
indefinitely.
|
Already in starting footprint; net future gains + floor |
| Sparsity, pruning & MoE |
Use fewer weights per token. Gemma 4’s MoE has 25.2B total
parameters and 3.8B active.6
|
Inactive experts still need storage and usually residency.
Routing, cache misses and uneven expert access limit ideal
speedups.
|
Active fraction changes traffic/compute, not total weights
|
| Attention & state architecture |
GQA, shared KV, sliding windows, recurrent/state-space or
hybrid designs can reduce context memory and traffic.11, 6
|
Long-context retrieval and exact recall must survive.
Efficient attention kernels do not remove the need to read
model weights.
|
Separate KV/state improvement; approximate attention term
|
| Speculative & multi-token decoding |
Draft several tokens, then verify. Gemma 4 includes matching
MTP drafters for local inference.24
|
The drafter takes RAM and compute; acceptance rates vary.
Algorithm/backend/device support matters.
|
Within effective throughput assumptions; no extra multiplier
|
| Reasoning, search & ensembles |
More inference-time work can solve harder problems, including
with smaller models.14
|
Hidden tokens, candidate rejection and parallel agents raise
latency, cache residency and joules per finished task.
|
Reasoning multiplier; concurrent agents excluded |
| Retrieval, tools & memory |
Local search, a calculator, code execution and personal
context improve the complete assistant.
|
A stronger system is not necessarily a stronger base model.
Fresh web data and cloud tools break full offline equivalence;
local indexes need storage.
|
Must match evaluation tools; additional app workload excluded
|
| Multimodal perception |
Efficient audio/vision encoders or unified architectures can
reduce processing stages.25
|
Camera/video input, audio output and continuous sensing add
tokens, compute and power; text-only fit is not full
multimodal fit.
|
Required in quality contract; text resource proxy only |
| Compute silicon & kernels |
NPUs, GPU matrix units, fused operators, compiler tuning and
cache locality improve usable arithmetic.5, 10
|
Peak TOPS mix precisions and sometimes sparsity conventions.
Unsupported operators can fall back to slower processors.
|
Sustained effective TOPS; prompt + decode compute gates |
| RAM capacity & bandwidth |
More affordable RAM fits weights; wider/faster memory moves
them. Packaging and larger caches can help both.
|
RAM capacity, bandwidth and flash storage are three separate
budgets. OS app limits and other tasks reduce availability.
|
Separate usable RAM and effective bandwidth curves |
| Battery, thermal design & duty cycle |
Lower joules per operation/byte and better cooling sustain
work longer.
|
Hot surroundings, a charging phone, battery aging and other
apps can lower the power budget; a peak burst is not a long
session.
|
Fixed AI power budget and declining energy coefficients |
| Economics & deployment |
Demand may justify more RAM, better packaging and
hardware/software co-design.
|
Bill of materials, manufacturing yield, DRAM supply, release
cycles and consumer replacement rates delay broad adoption.
|
Premium-device scenario; no mass-adoption forecast |
| Availability & reliability |
Open weights, local runtimes and good OS integration make
technically possible inference accessible.
|
Licenses, withheld weights, update policies and app
distribution can block deployment. Benchmarks can saturate or
miss rare, consequential failures.
|
External conditions; must verify before declaring arrival
|