DIP 1026---Deprecate Context-Sensitive String Literals---Community Review Round 1

[Dennis]

On Wednesday, 4 December 2019 at 01:26:24 UTC, H. S. Teoh wrote:
> And just for a bit more perspective, Python also has heredoc 
> syntax, so does Perl, PHP, bash, and probably many others. If 
> heredocs were really such a bad idea, why are people putting 
> them into so many languages, over and over again?

To me the opposite seems true. First of all:

> Python does not have here-docs. It does however have 
> triple-quoted strings which can be used similarly.
https://rosettacode.org/wiki/Here_document#Python

Then considering which notable languages have context-sensitive 
string literals:
1987: Perl
1989: Bash
1995: PHP
2001: D
2011: C++11

If you know any other examples, please tell. I don't think 
context-sensitive string literals were ever put in a notable 
language created after 2001. (C++ has the most recent addition, 
but parsing that is already so complex they have nothing to lose)

> That DIP seems dead in the water though. The author has 
> vanished and nobody has taken up the reins.

I was referring to Walter Bright's one:
https://github.com/dlang/DIPs/pull/165

> 1) Generating HTML snippets
> 2) Generating PovRay scene description snippets
> 3) Generating D code snippets
> 4) Generating snippets of a DSL I use for generating geometric 
> models
> 5) Generating boilerplate for input data to an external convex 
> hull
>    solver (has its own peculiar syntax)
> 6) Generating GLSL shader code snippets
> 7) Generating Java code snippets
> 8) Command line usage descriptions

I do believe for most of these you can use ``, q{} and q"<>" with 
little problems, but I understand that you prefer the q"EOS EOS" 
ones and would not want to rewrite your old code.

> ?!  Can't you just use a custom lexer with your PEG grammar?
>
> Then isn't the solution simply to write a self-contained 
> heredoc parsing function, put it in a dub package, and let 
> everyone reuse it? Then nobody will have to write it for 
> themselves again. Problem solved.

Of course you can make it work. I'm not saying that 
context-sensitive string literals make or break all D lexers, 
it's just a little source of complexity that may not bear its 
weight.

And a good couple of syntax highlighters support multiple 
different languages while being implemented in one, take for 
example this one written in Go:

https://github.com/alecthomas/chroma/blob/master/lexers/d/d.go

I wouldn't expect them to add dub package for D, cargo package 
for Rust, npm package for JavaScript etc.

> This whole debacle feels like heredocs are being singled out as 
> a scapegoat in a misguided quest to "simplify the language".  
> Like we're grasping at straws because we're unable to tackle 
> the bigger issues, so here's a convenient simple target we can 
> shoot and kill and feel good about ourselves that we're finally 
> making progress.  Talking about straining out the gnat and 
> swallowing the camel.

It seems to me D has this history of removing small features with 
a small problem:

- Small feature: escape string literals
   Small problem: doesn't have much use
- Small feature: octal string literals
   Small problem: can be confused for decimal literal, and can be 
made a library feature
- Small feature: hexstring literals
   Small problem: can be better represented in a library function

Now my proposed next one is:

- Small feature: context-sensitive string literals
   Small problem: accidentally bumps the complexity class of D's 
lexical grammar.

Now I understand that reviewers are debating whether it is a 
small feature ("I actually use these a lot") and whether the 
small problem isn't too small ("making D lexers still isn't 
hard"). That's what I like to see in the review, thanks 
especially to WebFreak and Adam D. Ruppe for their input on their 
VSCode and Vim highlighters, and thanks to you for your use cases.

What I don't get is why this is called a "non-starter" by Andrei 
and a "debacle" / "misguided quest" by you. Is it such a 
ludicrous idea to deprecate this particular part of the language?

I admit that I misjudged that amount of use, breakage and 
complexity this feature has before writing this DIP. If this 
trend continues then this DIP is dead, I'm not going to push this 
hard or anything. But I am at least still interested in Walter 
and Atila's opinion.