D Compiler as a Library

[Ary Manzana]

On 4/13/12 9:10 PM, deadalnix wrote:
> Le 13/04/2012 11:58, Ary Manzana a écrit :
>> Having a D compiler available as a library will (at least) give these
>> benefits:
>>
>> 1. Can be used by an IDE: D is statically typed and so an IDE can
>> benefit a lot from this. The features Descent had, as far as I remember,
>> were:
>> 1.1. Outline
>> 1.2. Autocompletion
>> 1.3. Type Hierarchy
>> 1.4. Syntax and semantic errors, showing not only the line number but
>> also column numbers if it makes sense
>> 1.5. Automatic import inclusion (say, typing writefln and getting a list
>> of modules that provide that symbol)
>> 1.6. Compile-time view: replace auto with the inferred type, insert
>> mixins into scope, rewrite operator overloads and other lowerings (but
>> I'm not sure this point is really useful)
>> 1.7. Determine, given a set of versions and flags, which branches of
>> static ifs are used/unused
>> 1.8. Open declaration
>> 1.9. Show implementations (of an interface, of interface's method or,
>> abstract methods, or method overrides).
>> 1.10. Propose to override a method (you type some letters and then hit
>> some key combination and get a list of methods to override)
>> 1.11. Get the code of a template when instantiated.
>> 2. Can be used to build better doc generators: one that shows known
>> subclasses or interface implementation, shows inherited methods, type
>> hierarchy.
>> 3. Can be used for lints and other such tools.
>>
>> As you can see, a simple lexer/parser built into an IDE, doc generator
>> or lint will just give basic features but will never achieve something
>> exceptionally good if it lacks the full semantic knowledge of the code.
>>
>> I'll write a list of things I'd like this compiler-as-library to have,
>> but please help me make it bigger :-)
>>
>> * Don't use global variables (DMD is just thought to be run once, so
>> when used as a library it can just be used, well, once)
>> * Provide a lexer which gives line numbers and column numbers
>> (beginning, end)
>> * Provide a parser with the same features
>> * The semantic phase should not discard any information found while
>> parsing. For example when DMD resolves a type it recursively resolves
>> aliasing and keeps the last one. An example:
>>
>> alias int foo;
>> alias foo* bar;
>>
>> bar something() { ... }
>>
>> It would be nice if "bar", after semantic analysis is done, carries the
>> information that bar is "foo*" and that "foo" is "int". Also that
>> something's return type is "bar", not "int*".
>> * Provide errors and warnings that have line numbers as well as column
>> numbers.
>> * Allow to parse the top-level definitions of a module. Whit this I mean
>> skipping function bodies. At least Descent first built a the outline of
>> the whole project by doing this. This mode should also allow specifying
>> a location as a target, and if that location falls inside a function
>> body then it's contents are returned (useful when editing a file, so you
>> can get the outline as well as semantic info of the function currently
>> being edited, which will never affect semantic in other parts of the
>> module). This will dramatically speed up the editor.
>> * Don't stop parsing on errors (I think DMD already does this).
>> * Provide a visitor class. If possible, use visitors to implement
>> semantic analysis. The visitor will make it super easy to implement
>> lints and to generate documentation.
>
> SDC have a lot of theses, and I proposed a similar stuff for its
> evolution. I think it is easier for SDC than it is for dmd considering
> the codebase of both.

Cool! SDC is the way to go. Let's focus our efforts on that project. :-)

One thing I saw in the code is that global variables are used in it 
(specially in the sdc.aglobal module).

Also, the lexer returns a TokenStream which contains the full array of 
tokens. This is very slow compared to returning them as they are 
scanned, when requested. But I guess this is easy to fix as it's hidden 
behind the TokenStream interface.

I tried to compile a simple file:

void main() {}

 > ./bin/sdc main.d
core.exception.AssertError at sdc.gen.sdcmodule(560): Assertion failure
----------------
5   sdc                                 0x000000010b87aa2a _d_assertm + 42
6   sdc                                 0x000000010b4e2daa void 
sdc.gen.sdcmodule.__assert(int) + 26
7   sdc                                 0x000000010b55d19c void 
sdc.gen.sdcmodule.Store.addFunction(sdc.gen.sdcfunction.Function) + 92
8   sdc                                 0x000000010b55d75b void 
sdc.gen.sdcmodule.Scope.add(immutable(char)[], 
sdc.gen.sdcfunction.Function) + 123
9   sdc                                 0x000000010b54ee36 void 
sdc.gen.declaration.declareFunctionDeclaration(sdc.ast.declaration.FunctionDeclaration, 
sdc.ast.sdcmodule.DeclarationDefinition, sdc.gen.sdcmodule.Module) + 1278
10  sdc                                 0x000000010b54e4b2 void 
sdc.gen.declaration.declareDeclaration(sdc.ast.declaration.Declaration, 
sdc.ast.sdcmodule.DeclarationDefinition, sdc.gen.sdcmodule.Module) + 146
11  sdc                                 0x000000010b54d12e void 
sdc.gen.base.genDeclarationDefinition(sdc.ast.sdcmodule.DeclarationDefinition, 
sdc.gen.sdcmodule.Module, ulong) + 374
12  sdc                                 0x000000010b54c8d0 void 
sdc.gen.base.resolveDeclarationDefinitionList(sdc.ast.sdcmodule.DeclarationDefinition[], 
sdc.gen.sdcmodule.Module, sdc.gen.type.Type) + 564
13  sdc                                 0x000000010b54c439 
sdc.gen.sdcmodule.Module 
sdc.gen.base.genModule(sdc.ast.sdcmodule.Module, 
sdc.aglobal.TranslationUnit) + 157
14  sdc                                 0x000000010b53432d int 
sdc.sdc.realmain(immutable(char)[][]) + 3081
15  sdc                                 0x000000010b4e283e _Dmain + 54
16  sdc                                 0x000000010b87b415 extern (C) 
int rt.dmain2.main(int, char**).void runMain() + 29
17  sdc                                 0x000000010b87adc5 extern (C) 
int rt.dmain2.main(int, char**).void tryExec(scope void delegate()) + 45
18  sdc                                 0x000000010b87b467 extern (C) 
int rt.dmain2.main(int, char**).void runAll() + 63
19  sdc                                 0x000000010b87adc5 extern (C) 
int rt.dmain2.main(int, char**).void tryExec(scope void delegate()) + 45
20  sdc                                 0x000000010b87ad48 main + 232
21  sdc                                 0x000000010b4e2694 start + 52
22  ???                                 0x0000000000000002 0x0 + 2
----------------

I couldn't find the line of the assertion. What are all those "+146" and 
"+54" about?