The D ecosystem in Debian with free-as-in-freedom DMD

[H. S. Teoh via Digitalmars-d]

On Mon, Apr 10, 2017 at 11:40:12AM +0000, Matthias Klumpp via Digitalmars-d wrote:
[...]
> Naturally, when the reference compiler is available in Debian, we
> would compile everything with that, as it is the development focus and
> the thing many people test with.
>
> We do, however, have quite a bit of bioinformatics and other D
> software in the archive where performance matters - so our users and
> the developers of that software (like BioD, potentially Mir, maybe
> even Vibe.d) will want the fastest performance and will ask us to
> compile the libraries with LDC or GDC.
> 
> If we do that, we will run into the D ABI trap: Libraries compiled
> with compiler X can not be used from software compiled with D compiler
> Y. There is actually no ABI stability guarantee even between DMD
> releases.  This will make integrating D a huge pain. Recompiling the
> dependency-chain of a software from source when compiling a package
> using the "right" compiler and statically adding the code is forbidden
> by distro policy.  Having static libraries in the dependencies doesn't
> solve the issue.  Compiling each library with all D compilers is
> highly impractical and not really feasible.

This is not a hard problem to solve, IMO.  Just build the library into
two separate binaries, each with a SONAME that encodes the ABI it is
compatible with.  The resulting two .so's can either be distributed as
separate packages (for minimum bloat, if that's a concern), or as a
single package that contains both binaries (since they have different
SONAMEs this should not be a problem).  Then if you compile some
software X that depends on this library, it will pick up the correct
version of the library depending on which compiler you compiled with.


> The other issue is architecture support. In Debian we are strongly
> encouraged to support as many architectures as possible, to the point
> of having to justify why arch X is not supported.

I'd say this makes a good case for my proposed solution above.

DMD is unlikely to support other archs than amd64/ia32 in the
foreseeable future, so the justification for dmd being unavailable for
arch X would be that upstream DMD simply doesn't support it.  This,
however, should not prevent us from using gdc/ldc on those other archs,
so that we can still ship packages for those archs.  They will merely
require ldc rather than dmd. And obviously, libraries built for that
arch will only support the ldc SONAME, not the dmd one. (This may be an
argument for bundling both SONAMEs in a single package -- it gets messy
if we start shipping multiple libraries, some of which may not be
available on all archs. By shipping a single package that includes both
versions for ia32/amd64, we can simply omit the DMD-compiled version
from other archs.)

Unfortunately, I realize that this means some packages that require the
latest DMD would not be available on all archs, if they require features
unavailable in gdc/ldc.  But this problem may not be a huge one, now
that ldc is mostly up-to-date with dmd (at most 1 release behind, IIRC).
GDC may lag behind a bit more because it is unfortunately tied to GCC
releases, so we may have to forego using gdc for building newer D
packages. But we should be able to ship most D packages compiled with
both.

Furthermore, I wonder if we should standardize on ldc for most D
software instead of dmd, unless that software absolutely depends on
features only available on dmd.  My reasons are:

- While dmd compiles very fast, it consistently (IME) produces code that
  runs about 20-30% slower than code produced by gdc (and I presume ldc
  as well). Since we're talking about building Debian packages on
  Debian's buildd's, which are background batch processes, compilation
  speed is no big deal, but the performance of the executable *is* a big
  deal. The last thing we want is to give a poor impression of D by
  shipping official Debian packages that have subpar performance.

- DMD is unlikely to target other archs than ia32/amd64 in the
  foreseeable future, AFAIK, unless the recent relicensing triumph of
  dmd's backend makes this future more likely.  There will definitely be
  resistance among DDs because of lack of support for other archs. LDC,
  in contrast, supports all of Debian's supported archs.

- LDC is already available in Debian, meaning that we can already start
  adding D packages built with ldc without needing to work through the
  red tape involved in adding a new compiler to the archive.

- The only case where I can see we'd want to compile with dmd is if the
  target software uses features that are only available on dmd.  Since
  LDC's release cycle is now very close to DMD's, I don't foresee there
  will be too many D projects out there that can't be compiled with LDC.
  So there should really only be a small number of packages that we'd
  have to fallback to dmd, which will also be limited in supported archs
  -- meaning that it will probably be challenging to get it into the
  Debian archive in the first place.

- If we standardize on using ldc to compile Debian packages, we can
  sidestep the ABI compatibility issue (though the SONAME hack will
  probably still be necessary in order to provide users with libraries
  that are usable with dmd).

Because of this, my first inclination is to standardize on ldc rather
than dmd, where building Debian packages are concerned.

Of course, this in no way diminishes the priority in *distributing* dmd
in the Debian archive as -dev packages. Just don't build other Debian
packages with it. :-D


[...]
> Also: If you want to help out Debian's D team, feel free to ping me or
> any other D team member (we are very short handed and are only two
> active people right now). See https://wiki.debian.org/D (caution, wiki
> page is very outdated, last touched in 2012)

I'd like to be involved, since I *am* technically a Debian developer
(though haven't been very active for the past long while -- I do try to
at least keep the meager few packages I maintain up-to-date, though).


T

-- 
"Holy war is an oxymoron." -- Lazarus Long