(Oh My) Gentool 0.3.0 released

[Gavin Ray]

On Friday, 7 May 2021 at 18:15:47 UTC, Jacob Carlborg wrote:
> On 2021-05-05 12:01, user1234 wrote:
>
>> Is it possible to use libclang and more generally LLVM c++ api 
>> [directly in D](https://dlang.org/spec/cpp_interface.html) or 
>> the Cpp interface is too limited ?
>> Was this an option, have you tried ?
>
> Yes, it's possible to use libclang. DStep [1] is using that and 
> it fully written in D. Although DStep cannot create bindings 
> for C++ yet so I cannot guarantee that using only libclang will 
> work for C++ code.
>
> [1] https://github.com/jacob-carlborg/dstep

I don't think using libclang is a good idea for C++. I think it 
works very well and is the ideal choice (due to lack of 
dependencies and infra) for C, but C++ has a nightmarish AST.

Have been investigating doing this with LibTooling using the C++ 
API, however a bit differently than has been done before (will 
explain in a minute below).

The reasoning is that, when I spoke with LLVM developers, and 
developers who had built successful codegen tooling for C++, 
every one of them cautioned against using libclang, said that 
they regretted the choice, and that the API and AST info it 
exposes is not sufficient.

> _I had a chance to ask Atila this same thing, last Beerconf, 
> and that was also his stance -- roughly that a large part of 
> the hurdles with dpp (beyond C++ being a nightmare to start) 
> were that it was built on libclang._

---

I have a large personal interest in building/helping to build a 
viable C++-header-to-D-`extern (C++)` generator because I think 
that is what would unlock a vast amount of potential + power for 
D. Quick, no-hassle direct bindings to any (or even most) C++ 
libraries and tools.

But I've been doing my homework both on how to approach this and 
what the state of C++ bindgen is in D. I've tried:

- D++
- Dstep
- Ohmygentool
- SWIG, with and without "Directors" feature enabled
- CPP2D

Of those, I've had the best success C++ using Ohmygentool by a 
fairly large margin. It can often do several-thousand line 
projects with non-trivial `#includes` mostly automatically.

---

**However, I had an idea which I haven't seen tried yet, and have 
been prototyping:**
- Using `cppyy` in Python (which uses `cling`) for runtime 
bindings to C++ and ability to write raw C++ code in Python 
strings and JIT compile it.

- Allow users to write "drivers"/"clients" in Python which do the 
codegen. Since Python isn't compiled, this means you can realtime 
tweak and visualize your output much faster than manually 
recompiling a C++ based LibTooling application.

I am thinking of some kind of API where you can declare rules 
using annotations for AST nodes above functions for handling 
them. Something like:
```py
class DCodegen:
     # "t" here is a LibTooling AST node and we can use all of 
Clang/LibTooling's AST API
     @rule(lambda t: t.is_pointer() or t.is_reference() and \
                     t.pointee().is_record_indirection())
     def input(cls, t, args):
         return f"{{interm}} = &{c_util.struct_cast(t, '{inp}')};"

     @rule(lambda t: t.is_pointer() or t.is_reference())
     def input(cls, t, args):
         raise ValueError("unsupported input pointer/reference 
type {}".format(t))
```

This would allow people to contribute or tweak the codegen to 
their liking very rapidly.

For distribution, it could be done in an Ubuntu Docker container 
that comes with LLVM and Python in it, and the scripts, then 
mapped to local filesystem for read access + also if you want to 
edit the `DCodegen` script or supply your own Python file as the 
driver.

---

**What do you all think of this idea?**

- Would love to hear feedback/opinions.
- I unfortunately have neither the D nor C++ expertise to 
properly write the translation rules, so it would take some 
help/collaboration from the community.

---

Here's an example of loading LibTooling in Python and building an 
AST from some code.
You can see the C++ object in the `print()` output in the top 
right:

- ![](https://i.imgur.com/FkSq0N8.png)
- ![](https://i.imgur.com/7rNbegn.png)