Inference-only research prototype
PairFold-KV
Fold two attention entries into a mean and signed residual. Keep the original softmax equation. Store less than half the cache. Change no model weights.
01 / The idea
Two tokens become one structured pair.
This is a change of coordinates, not token eviction. At full precision, it reproduces the original two-token softmax attention exactly.
(1 + 2) / 2
(1 − 2) / 2
a = q · mₖx = q · rₖa+x, a−xlogit₁ = a + x logit₂ = a − x
pair output = (w₁+w₂)mᵥ + (w₁−w₂)rᵥ
The mean says what the two entries share. The residual says how they differ. A future query can still choose between them through the sign and magnitude of x.
The identity itself is lossless. Our implementation becomes approximate only when it quantizes the mean and residual components.
There is no learned codebook and no fine-tuning step. PairFold is not VQ.
02 / The codec
Spend bits where attention notices.
Value residuals proved more sensitive than key residuals, so the selected format is deliberately asymmetric.
mₖINT8key meanrₖINT4key residualmᵥINT8value meanrᵥINT8value residualThe packed count includes four FP16 scales. The implementation also keeps the newest 16 tokens exact, which lowers the measured prompt reduction from the 2.21× asymptote to 2.13×.
03 / The kernel
Read the packed cache once.
The fast path is a fused AVX2 and OpenMP kernel. It never creates a decoded cache or a full attention matrix.
04 / Results
The slowdown disappeared on CPU.
The clean run used DistilGPT-2, a 512-token prompt, 64 generated tokens, 12 CPU threads, and five alternating baseline and PairFold measurements.
| Metric | Default | PairFold |
|---|---|---|
| Median decode speed | 7.75 TPS | 9.16 TPS |
| Prompt cache | 9.00 MiB | 4.23 MiB |
| Cache after 64 tokens | 10.13 MiB | 4.74 MiB |
| Top-1 agreement | reference | 32 / 32 |
| Top-5 overlap | reference | 96.88% |
Attention time, all six layers
05 / Interpretation
A useful result, with a small evidence base.
The fused experiment answers one question well. PairFold's algebra does not require a 30% decode penalty. It does not yet answer the GPU or frontier-model question.
What the test supports
- No model-weight changes are required.
- The unquantized pair identity matches ordinary softmax attention.
- Direct packed attention can remove the unpacking bottleneck.
- K4/V8 retained all 32 top-1 predictions in this sample.
What remains unknown
- The benchmark used one 82M-parameter model.
- The shared CPU produced noisy absolute TPS measurements.
- The speed baseline used FP32 while cache bytes use a BF16 deployment baseline.
- PairFold still has linear memory and compute in context length.