llvm-d

[Jens Mueller]

Moritz Maxeiner wrote:
> On Sunday, 17 March 2013 at 20:05:09 UTC, Jens Mueller wrote:
> >Deimos just provides a plain D interface for C libraries.
> >What do you mean?
> >You can do static or dynamic linking.
> >With dynamic linking it will be loaded at startup time by the
> >loader.
> >But you can also load it at runtime using some system API which I
> >assume
> >you do. I started ddl (http://jkm.github.com/ddl/ddl.html) which
> >does
> >work with Deimos to some extent. A limiting factor is compilation
> >speed.
> 
> Technically deimos provides extern(C) functions, which by themselves
> have - as  you said - to be linked statically/dynamically and cannot
> be loaded at runtime.
> Normally to load at runtime you need a function pointer to give the
> OS functions to load from the shared lib and afaict you're using
> ctfe to extract the functions from deimos and create those functions
> pointers from them to circumvent that problem - kudos by the way,
> never even thought about that possibility.
> Now to the real problem: When you're creating a shared lib on
> windows (32bit) it's going to be COFF, normally. But optlink can't
> link with that, only with OMF - which is one of the reason, I
> believe, why support for the MSLinker was added to windows 64bit
> versions of dmd. You can use import libs to make dmd on win32 work
> with COFF shared libs, but again, that is extra work on the user of
> llvm-d, which I do not want. The solution for me was simply to cut
> out the middle man reponsible for that hassle (optlink) from the
> process.

This makes perfect sense. I understand why that was your solution. But
it is limiting for non-Windows users.

> On another note:
> - Afaict ddl doesn't have OS X support, so I can't use it as a
> replacement for what I already have. If you want to add that to ddl
> you could just look at llvm/util/shlib.d in llvm-d, it's done there,
> missing support for one of the three majors is a big no-no for me.

I assumed OS X is Posix compliant. I never checked that though. Thanks
for the pointer.

> - Another problem would be that I want to keep things compatible
> with dub, which would become very difficult seing as ddl doesn't
> have support for dub at present.

No problem. I can add this. It is easy to do. I didn't do it because I
wasn't sure which package manager will be the "official" one. This
hasn't been settled as far as I know.

> - Moreover, ddl doesn't seem to have support for loading from a
> specific directory (such as "lib"), or at least I can't find it

I don't know how this is done on Windows. On Linux you just set the
LD_LIBRARY_PATH.

> - There are functions in the LLVM C API, which are not available in
> every LLVM shared lib: The target specific ones. That means that -
> depending on which targets where selected when compiling th shared
> lib - these functions may or may not be in the shared lib and trying
> to load them from the shared lib MUST NOT throw an exception. If
> loading fails it simply means the shared lib doesn't have support
> for that specific target; ddl however, always throws an exception
> when it can't load a symbol.

Uhh. I didn't know. Unfortunately I have very little experience with
the LLVM C API. But it is a very important use case for ddl and Deimos
in general.

> Only with those issues resolved in ddl would it be even feasible for
> me to do a split into say llvm-d + (deimos)llvm and have llvm-d
> depend on ddl.
> And even then llvm-d would depend on 2 projects (ddl and
> (deimos)llvm) instead of 0 like now and more dependencies for very
> little to no functionality increase is a very big argument against
> doing so for me.

I see. It's up to you. You know my preference.

> >Even though I understand your arguments. Technically, they are
> >different
> >projects. As you say built on LLVM's C API. One could use git
> >submodules
> >to ease building the higher level API.
> 
> By that same argument, however, you could also claim that the D
> runtime core should be split into more projects - one who only
> imports the necessary C functions from the libc, and the other who
> does all the D stuff.
> Not the best example, I know, but the point still stands: When the D
> OOP API is done, the llvm-d user should have no more need to use the
> C functions by themselves. Also LLVM A API functions often do
> strange things you wouldn't expect and the documentation is
> sometimes just simply wrong; using the LLVM C API from D with D's GC
> without something in between that takes care of the GC/NonGC gap is
> dangerous at best, I believe.

Probably. There are very few users for such a use case.
But if you would use LLVM's plain C interface in D users who have used
the API in C will feel home. I mean why expose the LLVM C API as an
addition. It certainly does no harm.

> >In sum you want to load libraries at runtime and have a higher
> >level
> >API. Both are fine. But I question that it is a good decision to
> >do the
> >loading at runtime for each C library.
> 
> Not that I advocate doing it for every C library, but if one really
> wants to what would be the real downside? The program startup time
> would suffer, of course, but with current computer architectures
> would it really be by an amount that actually hurts? And would the
> comfort of not having to worry about different library formats and
> linkers  on different OS's outweigh that? It's not like it is
> actually calculation-heavy stuff, only loading symbols. You could
> even parallelise it if it's more than a couple of hundred.

Usually, I use dynamic linking. I believe you would do the same if
optlink worked.

> >I would use Deimos bindings. I even started one myself.
> >https://github.com/jkm/deimos-llvm/
> >But I cannot devote much time on it.
> 
> That would make it one for and one somewhat against^^
> In all seriousness, though: Even if I were to split llvm-d into the
> two parts, I can't do it right now, as only the C API stuff exists
> at present, so splitting would leave me with one empty github
> repository right now.
> What I could do, however, is copying the C parts from llvm-d instead
> of splitting them. If I were to do that, I'd need to know how to get
> it in the deimos "D programming" group, though.

The process is usually pretty simple. But the documentation is lacking.
There are several pieces floating around. I can try writing one document
and try to push it to the official web site.

> To sum it up: I'd much rather make a copy of the C stuff from
> llvm-d, make it a deimos project, but leave llvm-d untouched then to
> do a split and deal with the consequences. It might seem
> contraproductive, but the actual effort to support two very similar
> projects with updates is less than to deal with problems arising
> from the split.

Do as you like it. I do not think this is a bad approach.

Jens