DIP 1026---Deprecate Context-Sensitive String Literals---Community Review Round 1

[H. S. Teoh]

On Tue, Dec 03, 2019 at 08:40:14PM +0000, Paul Backus via Digitalmars-d wrote:
> On Tuesday, 3 December 2019 at 18:34:22 UTC, H. S. Teoh wrote:
[...]
> > As Andrei said, heredoc string are trivial to parse because they are
> > essentially a single big token.  This should not pose any problem
> > for the parser at all.
> 
> By definition, a context-free grammar is defined in terms of a finite
> set of non-terminal symbols (i.e., tokens). [1] The set of all string
> literals is infinite. Therefore, either string literals are not
> tokens, or D's grammar is not context-free.
[...]

I think you're imposing a needlessly literal(!) interpretation of
context-free grammars.  For example, integer literals are also unbounded
(there is no largest integer, therefore the set of integer literals is
infinite). Does that mean that a calculator program that includes
integer literals in its grammar is not context-free?  I think that's a
preposterous application of the definitions.

As far as the grammar is concerned, all integer literals are the same
terminal symbol, because the grammar does not (need to) distinguish
between them.

Treating string (or any other) literals as non-tokens makes no sense
because they are not symmetric with non-string (or other) tokens, e.g.,
D tokens allow arbitrary whitespace between them, yet you cannot
arbitrarily insert whitespace into a string literal without changing its
semantics.


T

-- 
Time flies like an arrow. Fruit flies like a banana.