[Issue 1466] New: Spec claims maximal munch technique always works: not for "1..3"

[BCS]

Reply to d-bugmail at puremagic.com,

> http://d.puremagic.com/issues/show_bug.cgi?id=1466
> 
> Summary: Spec claims maximal munch technique always works:
> not
> for "1..3"
> Product: D
> Version: 1.020
> Platform: All
> URL: http://digitalmars.com/d/1.0/lex.html
> OS/Version: All
> Status: NEW
> Keywords: spec
> Severity: minor
> Priority: P3
> Component: www.digitalmars.com
> AssignedTo: bugzilla at digitalmars.com
> ReportedBy: deewiant at gmail.com
> A snippet from http://digitalmars.com/d/1.0/lex.html:
> 
> "The source text is split into tokens using the maximal munch
> technique, i.e., the lexical analyzer tries to make the longest token
> it can."
> 
> Relevant parts of the grammar:
> 
> Token:
> FloatLiteral
> ..
> FloatLiteral:
> Float
> Float:
> DecimalFloat
> DecimalFloat:
> DecimalDigits .
> . Decimal
> DecimalDigits:
> DecimalDigit
> DecimalDigit:
> NonZeroDigit
> Decimal:
> NonZeroDigit
> Based on the above, if a lexer encounters "1..3", for instance in a
> slice: "foo[1..3]", it should, using the maximal munch technique, make
> the longest possible token from "1..3": this is the Float "1.". Next,
> it should come up with the Float ".3".
> 
> Of course, this isn't currently happening, and would be problematic if
> it did. But, according to the grammar, that's what should happen,
> unless I'm missing something.
> 
> Either some exception needs to be made or remove the "DecimalDigits ."
> possibility from the grammar and the compiler.
> 

or make it "DecimalDigits . [^.]" where the ^ production is non consuming.