Model specs brag about 1M+ token context windows. Tempting to read that as “smarter” — or as “just dump everything in.” Should you?
Two gotchas hide behind the big number:
- Effective ≪ advertised. On reasoning-heavy work, accuracy sags as the window fills — and even the best models don’t escape it. On the live Context Arena MRCR long-context board, 2026’s top model (gpt-5.6-sol) still slides from ~99% accuracy to ~62% by 512K tokens; most land at 40–60% well before that. Newer models pushed the drop-off later, but never erased it. (The 2023 paper “Lost in the Middle” first flagged the effect.)
- And you pay for it every turn. Each turn re-runs the whole conversation, so the more back-and-forth piles up, the more every turn costs. KV/Prefix-caching softens the blow a bit — a stable context is charged at lower cost — but you still re-pay it every turn, and anything new is full price. A big window is a running cost, not a one-time buy.
Treat long context window as something to reach for when you genuinely need broad recall. Otherwise, a curated shorter context — and a fresh session when it fills — is likely to beat a stuffed one.
💡 Try this: In e.g. GitHub Copilot CLI you can switch between a “modest” 256k-400k context window or a 1M+ context window using the /model command (use the Tab key to switch).
🔗 Context Arena — live long-context (MRCR) leaderboard
🔗 Anthropic 1M context
Your take on huge context windows?
🟢 Useful for retrieval, not a brain upgrade
🟡 I mostly just fill the window
🔴 Bigger window = better, right? I always use the 1M+ window
⚪ Haven’t thought about it
How do you decide what context window you use?