DIP 1026---Deprecate Context-Sensitive String Literals---Community Review Round 1

[Dennis]

On Tuesday, 3 December 2019 at 18:34:22 UTC, H. S. Teoh wrote:
> So why the hate against heredoc strings?

I don't think you use the same terminology as the DIP so I might 
misinterpret this, but I have nothing against here documents. I'm 
glad D provides plenty of useful string literals for including 
text in source code, it's just that some of them are rarely used 
and bump up the complexity class of D's lexical grammar.

D has 6 types of string literals ("double quote" `back tick` r"r 
string" q{tokens} 	q"<brackets>" q”EOS ident EOS”) with 3 
encoding options (char, wchar, dchar).
For comparison, Java has one. C# has two + interpolated strings.
There is a DIP for adding interpolated strings to D.

People are mentioning how D keeps adding adding features and is 
on a road towards C++ complexity. There is precedent for removing 
barely used features (see e.g. octal, escape or hexstring 
literals  on https://dlang.org/deprecate.html).

And of course there are always users that remorse the removal of 
their favorite feature, but in the long run everyone benefits 
from a simpler language.

As for your use case of code generation, I'm having trouble 
relating to it. I happened to write some code generation 
algorithms myself recently, and could do fine with q{} strings 
for large templates and regular "" or `` string for small token 
parts like "switch(".

- Do you truly have 50,000 character string literals in your code 
base?
- Can't you use bracket delimited strings instead, q"<like this?>"
- If accidental early termination in huge string literals is a 
concern, even an identifier-delimited string isn't always safe. 
Can't you use an `import()` statement on an external text file?
- If those 50,000 characters are code and you value readability 
of it, isn't it a problem that there is no syntax highlighting in 
a q"EOS EOS" string?
- Can you maybe post an example of some of your q"EOS EOS" 
strings used for code generation?

> As for poor syntax highlighting as mentioned in the DIP, how is 
> that even a problem with the language?! It's a strawman 
> argument based on skewed data obtained from badly-written 
> lexers that don't actually lex D code correctly. It should be 
> the syntax highlighter that should be fixed, rather than 
> deprecate an actually useful feature in the language.

The thing is, these string literals simply can't be expressed in 
e.g. a PEG grammar. The D's grammar is one complexity class 
higher than needed just for this one relatively obscure string 
literal. Sure you can say "not our problem, those tooling authors 
just need to account for D's complexity", but I don't think that 
is useful for D's tooling ecosystem.

> Not to mention, the long list of projects at the end that will 
> need to be updated, which includes dmd itself BTW, looks like 
> strong evidence of good use of such string literals

dmd only uses them in the test-suite, same as libdparse.
I can spend some more time in the DIP exploring how other 
packages use them however.