Need advice on variable expansion behavior

[Petar]

Hi all,

Mathias LANG and me have been working on fixing a regression in 
Dub's variable expansion implementation [1]. Background:

In order to support use cases where a variable is used as a 
prefix or suffix in an identifier-like strings (e.g. 
build/${DUB_PACKAGE}_${DUB_PLATFORM}_${DUB_ARCH}), pull request 
[2] added support for syntax ${var}, in addition to the existing 
$var syntax.

Unfortunately that PR broke dollar sign escaping ($$) which was 
not caught by neither Dub's own unit tests, integration tests, 
nor any of the 46 Dub packages tested on Buildkite [4] (our CI 
for testing each pull request to dmd/druntime/phobos/tools/dub).

However, while testing the fix that Mathias originally wrote I 
bumped into a couple of edge cases and I realized the expected 
behavior is not thoroughly documented. As per the docs [4]:

> Inside of build setting values, it is possible to use variables 
> using dollar notation. Any variable not matching a predefined 
> name will be taken from the program environment. To denote a 
> literal dollar sign, use $$.

Should dollar signs not followed by a variable identifier be left 
as they are? What about escaped dollar signs ($$), or even $$$, 
$$$$, etc.? And what about $$$var?

For the purposes of this discussion, if we assume that each 
variable $var1 or ${var2} will be replaced by <VAR1> and <VAR2> 
respectively, which of the following outputs is the correct for 
the given input?

Source code:
https://run.dlang.io/gist/ZombineDev/eaf7076a7b83a556fac0c058b0525270?compiler=dmd

> Input:
> $ $$ $$$ $$$$ $$$$$ $$$$$$ 
> asd$$var1$var2$var3${var4}${var5}$${var6}$$var7$${var8} a$var9 
> b$$var10 c$$$var11 d$$$$var12 e$$$$$var13 f$$$$$$var14 
> g$$$$$$$var15 h$$$$$$$$var16 $$&$!$$?$$$$var18
>
> Petar#1:
> $ $$ $$$ $$$$ $$$$$ $$$$$$ 
> asd$var1<VAR2><VAR3><VAR4><VAR5>${var6}$var7${var8} a<VAR9> 
> b$var10 c$$var11 d$$$var12 e$$$$var13 f$$$$$var14 g$$$$$$var15 
> h$$$$$$$var16 $$&$!$$?$$$var18
> 
> Petar#2:
> $ $ $$ $$ $$$ $$$ 
> asd$var1<VAR2><VAR3><VAR4><VAR5>${var6}$var7${var8} a<VAR9> 
> b$var10 c$<VAR11> d$$var12 e$$<VAR13> f$$$var14 g$$$<VAR15> 
> h$$$$var16 $&$!$?$$var18
> 
> Mathias#1:
> $ $ $$ $$ $$$ $$$ 
> asd$var1<VAR2>$var3<VAR4>${var5}${var6}$var7${var8} a<VAR9> 
> b$var10 c$$var11 d$$var12 e$$$var13 f$$$var14 g$$$$var15 
> h$$$$var16 $&$!$?$$var18
> 
> Martin:
> $ $$ $$$ $$$$ $$$$$ $$$$$$ 
> asd$<VAR1><VAR2><VAR3><VAR4><VAR5>$<VAR6>$<VAR7>$<VAR8> a<VAR9> 
> b$<VAR10> c$$<VAR11> d$$$<VAR12> e$$$$<VAR13> f$$$$$<VAR14> 
> g$$$$$$<VAR15> h$$$$$$$<VAR16> $$&$!$$?$$$<VAR18>
> 
> Original:
> <> $ $<> $$ $$<> $$$ 
> asd$var1<VAR2><VAR3><>{var4}<>{var5}${var6}$var7${var8} a<VAR9> 
> b$var10 c$<VAR11> d$$var12 e$$<VAR13> f$$$var14 g$$$<VAR15> 
> h$$$$var16 $&<>!$?$$var18
> 
> Other: ???

Cheers,
Petar

[1]: https://github.com/dlang/dub/pull/1552
[2]: https://github.com/dlang/dub/pull/1392
[3]: 
https://gist.github.com/ZombineDev/fd79c4ffd58dbe9792d799f66b7f9434
[4]: 
https://code.dlang.org/package-format?lang=json#environment-variables