← Lab

Does encrypted compaction use fewer tokens than text summaries?

Result

Encrypted compaction used 22% fewer GPT-6 Luna tokens and cost 10% less per run than text summaries.

Tokens used per coding run

  • Text summary1.18M
  • Encrypted state0.91M
Mean token use across scored coding runs, including the calls used to summarize context.

Pass rate on tasks that often compacted

  • Text summary56.9%
  • Encrypted state53.5%
The pass-rate difference was −3.5 points, with a 95% interval from −15.3 to +6.9. Neither method established a quality win.

Methodology

Run GPT-6 Luna at high reasoning on 24 Terminal-Bench 2.1 coding tasks, four times per method in fresh containers. At 50000 context tokens, either replace older context with a text summary or use OpenAI’s encrypted compacted state. Pair the two methods on the same tasks and exclude infrastructure failures. Compare token use across all scored runs; compare pass rates on the 12 tasks that frequently compacted, giving each task equal weight.

← All findings