[Issue 18498] New: File buffer overrun when using lexer-as-library

[d-bugmail at puremagic.com]

https://issues.dlang.org/show_bug.cgi?id=18498

          Issue ID: 18498
           Summary: File buffer overrun when using lexer-as-library
           Product: D
           Version: D2
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P1
         Component: dmd
          Assignee: nobody at puremagic.com
          Reporter: bugzilla at digitalmars.com

As reported on the internet:
---
Reminds me of a bug I ran into recently while using the dmd lexer in a project.

When I passed any valid file to the lexer it worked fine.

When I passed a batch of valid files to the lexer, after a few files were
processed, it started to spit out all kinds of errors about invalid Unicode
code-points and ghost tokens. All those locations were past EOF.

When I passed in that particular file individually it worked like a charm.

When I over-sized the buffer that held a file by one char and set that to 0 it
worked for the batch of files.

--