How can we make it easier to experiment with the compiler?

[Nicholas Wilson]

On Monday, 24 May 2021 at 02:56:14 UTC, Walter Bright wrote:
> On 5/23/2021 7:25 PM, Nicholas Wilson wrote:
>> Directories.
>
> The #1 problem isn't directories, it's "every module imports 
> every other module" that leaves one with nowhere to start.
>
> We currently have:
>
>   dmd
>   dmd/root
>   dmd/backend
>
> I regularly fend off attempts to have dmd/root import files 
> from dmd, and dmd/backend import files from dmd. I recently had 
> to talk someone out of having dmd/backend import files from 
> dmd/root.
>
> In other words, a failure of encapsulation.

This is a _completely_ orthogonal problem.

The symptoms are completely orthogonal, although easily confused: 
failure of encapsulation makes _reasoning_ about the 
_interconnectedness_ of code difficult, failure to package makes 
_exploration_ and _enumeration_ of code (files, functions, 
classes, data structures) more difficult.
The solutions, however are cross enabling: we can implement and 
_enforce_ policies like say "AST node implementing modules should 
not import semantic analysis modules" with reasonable confidence 
iff we have all the AST modules in one place and all the semantic 
analysis modules in one place.

The symptoms of failure of encapsulation I'm going to assume you 
are well aware of.
The symptoms of failure to use packages are as follows:
  * the sheer number of filed in src/dmd make it impossible to 
remember what each file is for.  This problem is compounded by 
the fact that many files have names that do not describe well 
what they do _especially_ to newcomers. Principle offending 
example `ob`. Compare with names like `filecache`.
* it is impossible to determine at a glance what files are 
related to each other:
is `foreachvar.d` an AST node?, what about `dcast.d`? (No and No)
Whats the difference between `glue.d` and `gluelayer.d`?
is `visitor.d`, `transitivevisitor.d`, `strictvisitor.d` 
`parsetimevisitor.d` and `permissivevisitor.d` a complete list of 
the module public visitor modules? (No)
Which of `cond.d` and `staticcond.d` is the AST node for a static 
condition? What does the other file do? (`cond.d`, semantic 
analysis)
What files do semantic analysis? Which files declare AST nodes? 
Which files interface with the backend (and subsequently are not 
part of LDC or GDC)?
Where is DMD's entry point?


> snip example
> Fixing this is not easy, it requires a lot of hard thinking 
> about what a module *really* needs to do. But each success at 
> eliminating an import makes it more understandable.

Fixing the lack of directory issue requires only to think about 
what a module _is_ i.e. what package it belongs to: 
driver/frontend (mars, errors etc) , lexer group (lex, parse, 
tokens etc), ast, semantic analysis, backend interfacing, 
backend, root.

> Creating a false hierarchy (an implied relationship that is 
> instantly defeated by the imports)

You cannot seriously tell me with a straight face that e.g. AST, 
is not a hierarchy and should not be grouped together.

> of files won't fix [failure to encapsulate].

Indeed is fixes a different problem, but it makes fixing failure 
to encapsulate much easier.

> A good rule of thumb is:
>
>     *** Never import a file from an uplevel directory ***
>
> Import sideways and down, never up.

Indeed. However you can't to much of that with just

>   dmd
>   dmd/root
>   dmd/backend