[spec] Phases of translation

[Dibyendu Majumdar]

On Monday, 20 May 2019 at 22:17:07 UTC, Dibyendu Majumdar wrote:
> On Monday, 20 May 2019 at 16:13:39 UTC, Walter Bright wrote:
>
>> The crucial thing to know is that the tokenizing is 
>> independent of parsing, and parsing is independent of semantic 
>> analysis.
>>
>
> I am trying to understand this aspect - I found the small write 
> up in the Intro section not very clear. Would be great if you 
> could so a session on how things work - maybe video cast?
>
> For example, the mixin declaration has to convert a string to 
> AST I guess? When does this happen? Does it not need to invoke 
> the lexer on the generated string and build AST while already 
> in the semantic stage?
>

Currently the Intro says:

The process of compiling is divided into multiple phases. Each 
phase has no dependence on subsequent phases. For example, the 
scanner is not perturbed by the semantic analyzer. This 
separation of the passes makes language tools like syntax 
directed editors relatively easy to produce. It also is possible 
to compress D source by storing it in ‘tokenized’ form.

I feel this description is unclear, and it might just reflect how 
DMD is implemented. I haven't implemented a C++ parser but 
parsers I have worked with - such as for C - it is always the 
case that lexer doesn't get impacted by the semantic analysis. 
The standard process is to get a stream of tokens from the lexer 
and work with that. It is also conceivable that someone could 
create a "dumb" AST first for C++, and as a subsequent phase add 
semantic meaning to the AST, just as is done for D.

For now I propose to remove this paragraph until there is a 
better description available. Please would you review my pull 
request as it it is blocking me from doing further work.

Thanks and Regards
Dibyendu