GDMD D port ready for alpha-testing

[Johannes Pfau via D.gnu]

Am Sun, 27 Mar 2016 08:35:00 +0000
schrieb Vladimir Panteleev <thecybershadow.lists at gmail.com>:

> Thanks for working on this :)
> 
> On Saturday, 26 March 2016 at 22:37:53 UTC, Johannes Pfau wrote:
> > I didn't realize response file parsing is that important for 
> > windows, thanks for letting me know ;-) I guess there's no 
> > proper documentation for response files or any test files?  
> 
> It uses the CommandLineToArgvW syntax.
> 
> That's pretty much all there is to it.
> 
> > There are some open questions:
> >
> > * Are escape sequences allowed outside of quoted strings?  
> 
> Apparently, though rdmd will never generate such response files.
> 
> > * Which characters end a comment?  
> 
> There are no comments in response files (as far as I know).
> [...]

Well, that's the problem: There's no specification for response files.
The DMD implementation certainly does support comments in response
files. LDMD simply reuses the DMD code because of these issues...

The entries in the response file are formatted according to
CommandLineToArgvW, this part is simple.

But the entries are separated by 'whitespace' which according to dmd is
'\r' '\n' '\t' ' ' and '\0'! This leads to many questions such as does
'\0' end a comment? Additionally dmd treats '#' as the start of a
comment and 0x1a always ends the file...

DMD also only does the '\' escaping for double quotes, not for single
quotes. I think the code I've pushed should be as DMD-compatible as
possible.

> I'm not sure this approach is the best - I think unescaping and 
> splitting should be done in one pass. Essentially, I think 
> arguments are separated by an unescaped space.

Sure, escaping needs to be considered when splitting the string (to
differentiate between whitespace in quoted strings / whitespace and to
differentiate escaped quotes from normal quotes). But the code is nicer
if the splitting pass simply returns a slice to the original content
and does not assemble the unescaped strings on the fly.

> If you have a Windows machine, you can verify it yourself by 
> comparing the results with CommandLineToArgvW.
> 
> > https://github.com/D-Programming-GDC/GDMD/commit/0e2b0744a13a69f9dae90cf5db8c689ccca82f10
> > It'd be great if you could have a look at the unittests and 
> > tell me if
> > the parsing is correct ;-)  
> 
> The unit tests *look* OK, but I can't say for sure without having 
> a way to verify them.
> 
> I suggest you have a look at the escaping code and tests from 
> std.process. There should be no licensing conflict with Boost, I 
> think. I went to great lengths to ensure that code is correct, 
> there is even a test which brute-forces special character 
> combinations and another that tries random strings to find bugs 
> in the escaping code.
> 
> IIRC the simple regex that was in the Perl version of gdmd was 
> also sufficient for parsing the response files generated by rdmd.
> 

Yeah, probably supporting the sane subset of response files actually
used is good enough. I tried to match the dmd implementation, so any
weird special feature works, but maybe that's not required.

> > Now I still have to generate a response file for passing 
> > arguments to GDC. First have to check whether GCC even uses the 
> > windows escaping rules though. Do you know how many characters 
> > can be passed without response files?  
> 
> rdmd uses a limit of 1024, but I think that's lower than the OS 
> limit.
> 

OK, thanks!