[spec] Phases of translation

[Dibyendu Majumdar]

On Tuesday, 21 May 2019 at 13:24:11 UTC, Dibyendu Majumdar wrote:
>> On Monday, 20 May 2019 at 16:13:39 UTC, Walter Bright wrote:
>>
>>> The crucial thing to know is that the tokenizing is 
>>> independent of parsing, and parsing is independent of 
>>> semantic analysis.
>>>
>>
>> I am trying to understand this aspect - I found the small 
>> write up in the Intro section not very clear. Would be great 
>> if you could so a session on how things work - maybe video 
>> cast?
>>
>> For example, the mixin declaration has to convert a string to 
>> AST I guess? When does this happen? Does it not need to invoke 
>> the lexer on the generated string and build AST while already 
>> in the semantic stage?
>>
>
> Currently the Intro says:
>
> The process of compiling is divided into multiple phases. Each 
> phase has no dependence on subsequent phases. For example, the 
> scanner is not perturbed by the semantic analyzer. This 
> separation of the passes makes language tools like syntax 
> directed editors relatively easy to produce. It also is 
> possible to compress D source by storing it in ‘tokenized’ form.
>
> I feel this description is unclear, and it might just reflect 
> how DMD is implemented. I haven't implemented a C++ parser but 
> parsers I have worked with - such as for C - it is always the 
> case that lexer doesn't get impacted by the semantic analysis. 
> The standard process is to get a stream of tokens from the 
> lexer and work with that. It is also conceivable that someone 
> could create a "dumb" AST first for C++, and as a subsequent 
> phase add semantic meaning to the AST, just as is done for D.
>
> For now I propose to remove this paragraph until there is a 
> better description available. Please would you review my pull 
> request as it it is blocking me from doing further work.
>

Hi Walter, please would you share any insights regarding above?

Thanks and Regards
Dibyendu