DConf 2014 Day 1 Talk 4: Inside the Regular Expressions in D by Dmitry Olshansky

Dmitry Olshansky via Digitalmars-d-announce digitalmars-d-announce at puremagic.com
Thu Jun 12 09:42:17 PDT 2014


12-Jun-2014 16:25, Dicebot пишет:
> On Thursday, 12 June 2014 at 09:17:45 UTC, Dmitry Olshansky wrote:
>> This one thing I'm loosing sleep over - what precisely is so good in
>> CTFE code generation in _practical_ context (DSL that is quite stable,
>> not just tiny helpers)?
>>
>> By the end of day it's just about having to write a trivial line in
>> your favorite build system (NOT make) vs having to wait for a couple
>> of minutes each build hoping the compiler won't hit your system's
>> memory limits.
>
> Oh, this is a very good question :) There are two unrelated concerns here:
>

It's always nice to ask something on D NG, so many good answers I can 
hardly choose whom to reply ;) So this is kind of broadcast.

Yes, the answer seems spot on - reflection! But allow me to retort.

I'm not talking about completely stand-alone generator. Just as well 
generator tool could be written in D using the same exact sources as 
your D program does. Including the static introspection and 
type-awareness. Then generator itself is a library + "an invocation 
script" in D.

The Q is specifically of CTFE in this scenario, including not only 
obvious shortcomings of design, but fundamental ones of compilation 
inside of compilation. Unlike proper compilation is has nothing 
persistent to back it up. It feels backwards, a bit like C++ TMP but, of 
course, much-much better.

> 1)
>
> Reflection. It is less of an issue for pure DSL solutions because those
> don't provide any good reflection capabilities anyway, but other code
> generation approaches have very similar problems.
>
> By doing all code generation in separate build step you potentially lose
> many of guarantees of keeping various parts of your application in sync.
>

Use the same sources for the generator. In essence all is the same, just 
relying on separate runs and linkage, not mixin. Necessary "hooks" to 
link to later could indeed be generated with a tiny bit of CTFE.

Yes, deeply embedded stuff might not be that easy. The scope and damage 
is smaller though.

> 2)
>
> Moving forward. You use traditional reasoning of DSL generally being
> something rare and normally stable. This fits most common DSL usage but
> tight in-language integration D makes possible brings new opportunities
> of using DSL and code generation casually all other your program.
>

Well, I'm biased by heavy-handed ones. Say I have a (no longer) secret 
plan of doing a next-gen parser generator in D. Needless to say swaths 
of non-trivial code generation. I'm all for embedding nicely but I see 
very little _practical_ gains in CTFE+mixin here EVEN if CTFE wouldn't 
suck. See the point above about using the same metadata and types as the 
user application would.

> I totally expect programming culture to evolve to the point where
> something like 90% of all application code is being generated in typical
> project. D has good base for promoting such paradigm switch and reducing
> any unnecessary mental context switches is very important here.
>
> This was pretty much the point I was trying to make with my DConf talk (
> and have probably failed :) )

I liked the talk, but you know ... 4th or 5th talk with CTFE/mixin I 
think I might have been distracted :)

More specifically this bright future of 90%+ concise DSL driven programs 
is undermined by the simple truth - no amount of improvement in CTFE 
would make generators run faster then optimized standalone tool 
invocation. The tool (library written in D) may read D metadata just fine.

I heard D builds times are important part of its adoption so...

>
>> And these couple of minutes are more like 30 minutes at a times. Worse
>> yet unlike proper build system it doesn't keep track of actual changes
>> (same regex patterns get recompiled over and over), at this point
>> seamless integration into the language starts felling like a joke.
>>
>> And speaking of seamless integration: just generate a symbol name out
>> of pattern at CTFE to link to later, at least this much can be done
>> relatively fast. And voila even the clunky run-time generation is not
>> half-bad at integration.
>>
>> Unless things improve dramatically CTFE code generation + mixin is
>> just our funny painful toy.
>
> Unfortunately current implementation of frontend falls behind language
> capabilities a lot. There are no fundamental reasons why it can't work
> with better compiler.

It might solve most of _current_ problems, but I foresee fundamental 
issues of "no global state" in CTFE that in say 10 years from now would 
look a lot like `#include` in C++. A major one is there is no way for 
compiler to not recompile generated code as it has no knowledge of how 
it might have changed from the previous run.

> In fact, deadlnix has made a very good case for
> SDC taking over as next D frontend exactly because of things like CTFE JIT.

Yeah, we ought to help him!

-- 
Dmitry Olshansky


More information about the Digitalmars-d-announce mailing list