On the D Blog: Language Plasticity is More Important Than Ever
Adam Wilson
flyboynw at gmail.com
Mon Aug 10 22:35:46 UTC 2026
On Monday, 10 August 2026 at 13:41:59 UTC, monkyyy wrote:
> If that where the numbers come from, they are suggesting
> deleting whitespace; grouping 4-spaces as 1 token is a "gpt-2"
> level technology.
There has been a lot of work done at the harness and ingestion
layers to reduce input token usage precisely because
whitespace-sensitive languages are so token hungry. So yes,
they've gotten the input token usage down, but by improving
tooling, not because the language itself is less token expensive.
And there is evidence, albeit in early stages, that such schemes
are increasing the amount of mistakes that the LLM makes on
output and thus creating more iterations over the problem. Which
increases total token usage.
More information about the Digitalmars-d-announce
mailing list