foreach over split string

[JS]

On Wednesday, 17 July 2013 at 23:56:11 UTC, H. S. Teoh wrote:
> On Thu, Jul 18, 2013 at 01:02:26AM +0200, JS wrote:
> [...]
>> I can't put together a working example right now(other things 
>> to do)
>> but the gist of the matter is:
>> 
>> template strsplit(string n) { enum strsplit = 
>> std.string.split(n,
>> ","); }
>> 
>> ... inside a ctfe ...
>> 
>> foreach(n; strsplit!(s)) // doesn't work
>> foreach(n; std.string.split(s, ",")) // works
>> 
>> To me, this is confusing as hell and "n can't be read at 
>> compile
>> time" took me a good few hours to realize it was simply using 
>> the
>> template.  I'm guessing that it has something to do with the 
>> enum
>> "return type" of strsplit but I've tried other various things.
>
> I think it will help to understand the difference between 
> templates and
> CTFE. Even though both are processed at compile-time, they are 
> not the
> same thing, and mixing them may not always do what you think 
> it's doing.
>
> Basically, a template is used for generating code, so any 
> template
> parameters must be known "at compile-time", which means that it 
> must
> ultimately reduce to a typename, or a literal of a built-in 
> type. Note
> that this is NOT the same thing as "this variable is being used 
> by CTFE
> so the compiler should know what its value is at 
> `compile-time'". At
> this conceptual stage of compilation, the compiler hasn't fully
> transformed the source code into runnable form yet, so the only 
> thing it
> can do at this point is to work with, conceptually-speaking, 
> text-level
> entities. It can't handle variables and other such things that 
> only
> exist when a program has been transformed into a runnable form 
> and
> executed by an interpreter.
>
> CTFE is basically a subsystem of the compiler that has the 
> ability to
> run code *after* it has been generated (either via parsing the 
> source
> code directly, or via expanding templates). That is to say, the 
> program
> is now in a (semi)runnable form, but the compiler hasn't output 
> the
> object files / executables yet. Basically kinda like an "early
> execution" of your program, if you will. Since this happens in a
> conceptually later stage than template expansion, whatever's 
> going on in
> CTFE generally can't be fed back into template parameters. 
> Template
> expansion must have been completed before CTFE even comes into 
> play.
>
> I think part of the confusion may stem from the liberal use of 
> the term
> "compile-time", which may have given a false impression that if 
> entity X
> is known "at compile-time", then it should be usable by 
> construct Y
> which requires stuff that's known "at compile-time". This 
> oversimplifies
> the process of compilation; in reality, "compile-time" consists 
> of
> several distinct conceptual phases (in implementation this may 
> not
> necessarily be true literally, but it helps to understand the 
> conceptual
> stages). What generally happens is that the compiler needs to:
>
> 1) Read the text representation of the source code, break that 
> down
> into tokens ("lexing"), and based on the sequence of tokens, 
> construct
> an abstract syntax tree that represents the structure of the 
> source code
> ("parsing");
>
> 2) Translate the syntax tree into an internal representation 
> (IR) of the
> program that more-or-less maps to the machine code that will 
> eventually
> be output;
>
> 3) Translate the IR into actual instructions the target CPU can
> understand, and output that as the object file / executable.
>
> Conceptually-speaking, template expansion comes somewhere 
> between (1)
> and (2): it's a mechanism for producing large numbers of 
> very-similar
> subtrees of the syntax tree without requiring the programmer to 
> do lots
> and lots of typing. CTFE is roughly somewhere between (2) and 
> (3): the
> program isn't fully compiled yet, but enough translation has 
> been done
> that it is now in a form that can actually be run (by an 
> interpreter).
> In order to get to this point, template expansion must have been
> completed first, since otherwise your syntax tree isn't 
> complete yet, so
> by definition it can't have been translated into a runnable 
> form yet.
> But since template expansion must have already taken place, 
> that means
> you can't feed CTFE values back into templates -- since 
> otherwise
> template expansion couldn't have already taken place!
>
> So, long story short, template parameters must be known at 
> "text-level",
> if we can coin a term for that, the compiler must be able to 
> reduce said
> parameters into a concrete type name or a literal of a built-in 
> type.
> CTFE, on the other hand, works after template expansion has 
> completed
> (conceptually speaking), and so is essentially a kind of "early
> execution" of your program. CTFE variables and the like are
> "interpreter-level", so they can't be passed back into template
> parameters that expect "text-level" input.
>
> The actual details of what the compiler does is, of course, 
> quite a bit
> more complex than the above (over)simplified description, but 
> generally
> speaking, if you stick by the rule that CTFE values can't be 
> fed back
> into the templating system, you will experience much less 
> frustration.
>
>
> T


Thanks, this has made it much clearer.

Something like

foreach(a; StrSplit!(s))
    foreach(b; StrSplit !(a))

does work because the second StrSplit uses a "ctfe-time variable" 
instead of a "template-time variable".

My logic was:

1. first StrSplit resolved
2. first foreach evaluated
3. second StrSplit resolved
4. second foreach evaluated

while it actually is

1. first StrSplit resolved
2. second StrSplit resolved
3. foreach's resolved

because template expansion happens before any ctfe expansion.

I guess I was thinking the compiler would be smart enough to 
interleave template expansion and ctfe code(which would be much 
more powerfull).

Effectively template expansion is a sort of pre-processing to 
ctfe code and must be static as far as ctfe's go.

Does this make sense?