On the D Blog: Language Plasticity is More Important Than Ever
monkyyy
crazymonkyyy at gmail.com
Mon Aug 10 13:41:59 UTC 2026
On Monday, 10 August 2026 at 12:57:42 UTC, Mike Parker wrote:
> In this blog post, Adam Wilson goes into the importance of
> language plasticity in the LLM era, particularly as it applies
> to the number of tokens used on a given task, and why D has an
> advantage.
>
> https://blog.dlang.org/2026/08/10/language-plasticity-is-more-important-than-ever/
> The tabs-versus-spaces debate has long divided developers. In
> the LLM era, it carries real financial weight. Many tokenizers
> treat whitespace as distinct tokens, and common conventions
> amplify the difference.
> In practice, this means every level of nesting can add multiple
> individual space tokens
> This flexibility lets teams choose tabs (a single character)
> without breaking semantics.
from your own first source; the first image
https://arxiv.org/html/2508.13666v1/x1.png
No; those grouped yellow-highlighted spaces is suggesting they
are encoding them together
If that where the numbers come from, they are suggesting deleting
whitespace; grouping 4-spaces as 1 token is a "gpt-2" level
technology.
More information about the Digitalmars-d-announce
mailing list