std.d.lexer requirements

[Piotr Szturmaj]

Walter Bright wrote:
> On 8/2/2012 1:26 PM, Jonathan M Davis wrote:
>> On Thursday, August 02, 2012 01:44:18 Walter Bright wrote:
>>> Keep a 6 character buffer in your consumer. If you read a char with
>>> the high
>>> bit set, start filling that buffer and then decode it.
>>
>> And how on earth is that going to work as a range?
>
> 1. read a character from the range
> 2. if the character is the start of a multibyte character, put the
> character in the buffer
> 3. keep reading from the range until you've got the whole of the
> multibyte character
> 4. convert that 6 (or 4) character buffer into a dchar

Working example: 
https://github.com/pszturmaj/json-streaming-parser/blob/master/json.d#L18 :-)