std.d.lexer requirements

[H. S. Teoh]

On Wed, Aug 01, 2012 at 09:06:42PM -0700, Jonathan M Davis wrote:
> On Wednesday, August 01, 2012 21:00:53 Walter Bright wrote:
> > On 8/1/2012 7:04 PM, deadalnix wrote:
> > > Le 02/08/2012 02:10, Walter Bright a écrit :
> > >> 6. A single "Lexer" instance should be able to serially accept
> > >> input ranges, sharing and updating one identifier table
> > > 
> > > I see the lexer as a function that take an range of char as input
> > > and give back a range of token. Does it make sense to make an
> > > instance of a lexer ?
> > Yes, as the lexer will have some state and some configuration
> > parameters.
> 
> Couldn't that just be part of the range type? A separate lexer type
> shouldn't be necessary.
[...]

Whether it's part of the range type or a separate lexer type,
*definitely* make it possible to have multiple instances. One of the
biggest flaws of otherwise-good lexer generators like lex and flex
(C/C++) is that the core code assumes a single instance, and
multi-instances were glued on after the fact, making it a royal pain to
work with anything that needs lexing multiple things at the same time.


T

-- 
The day Microsoft makes something that doesn't suck is probably the day
they start making vacuum cleaners... -- Slashdotter