D for Game Development

Manu via Digitalmars-d digitalmars-d at puremagic.com
Sun Aug 9 04:38:53 PDT 2015


On 9 August 2015 at 15:31, Walter Bright via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
> On 8/8/2015 7:40 PM, Manu via Digitalmars-d wrote:
>>
>> 1. DMD has unsatisfactory codegen for anything other than debug builds.
>
>
> Do you mean the codegen is slower? But consider that the bottleneck in most
> programs is a small section of code. Taking a good look at the generated
> code for that and comparing with another compiler can often hint at an easy
> improvement to dmd that can address that bottleneck.
>
> But waiting for someone else to discover the same thing on some other piece
> of code means you'll be waiting a long time.

I understand, but that's not sustainable.
We can't be in a situation where I'm at work with a deadline, and I
need to invoke the community to address a codegen issue that's
blocking a release. Unexpected surprises like that with unknown/long
turn-around times aren't a practical commitment.
Last time I was concerned with this, x87 was destroying DMD
performance. Floats were x87, moved to XMM (according to x64 ABI) when
passed to functions, then moved back into x87 on the other side when
they're operated on.
Float code was overwhelmed with shuffling values between registers.
Basically all hot code is float code in my line of work.

I think Ethan was in discussions with you about this some time back?
It was a problem at the time I left Remedy.

Also, I think one of the big issues is the optimiser in general. The
inliner's just not fantastic, and D depends on it perhaps even more
than C++, especially when using the modern lazy code style.
I'm sure I can hand-assist DMD to produce the assembly I expect for
some hot function with some work, but traditional hand-assisted code
generation all falls apart when you start stacking lazy-ranges and
generally using the modern D style.
DMD's backend has almost no development, it's advancing at near-zero
velocity compared to LLVM. We need to get out of the situation where
LDC and GDC are perpetually 1-2 versions behind.

Incidentally, In my recent (dead-end) experience with LDC, I get
endless ICE's whenever I try to do anything mildly complex. When you
have your Debug and Release build straddling language versions and
compilers with different ICE characteristics, it's just not a
practical workflow. Only green-fields projects or the simplest of
simple apps can hope not to trigger bugs constantly in that
environment.


>> 2. DMD generates x87 code, and uses real everywhere.
>
>
> Less so now than it used to. For float and double, it uses SIMD.

Okay, is this fairly recent? My last set of comprehensive tests are
probably out of date.


>> We can't be
>> generating new x87+real instructions in 2015. It's deprecated
>> hardware!
>
>
> x87 works on every x86 CPU, and I doubt it will ever go away, deprecated or
> not. Why was it a problem for you?

Massive register swapping was the issue. I'd like to think register
swapping should never be implicitly generated unless I *explicitly*
type the real keyword, I don't want to see x87 opcodes.


>> 2. LDC has a lot more bugs than DMD (which still has quite a few
>> bugs), and the Windows build is new and even more immature still.
>> 3. LDC has no debuginfo. **biggest practical issue by far!**
>> 4. LDC/GDC are always a few versions behind DMD. This creates an
>> awkward/almost-impossible situation when you rely on DMD to build
>> debug code, and LDC to build releasable code. Depending on 2 flaky
>> compilers is even less fun than one.
>> 5. DLL's were really flakey at the time, I'm not sure how they are
>> now, but I'm concerned they may not be much better.
>> 6. Shared druntime; we were loading D code from DLL's, lots of them.
>> It's not really reasonable for each DLL to have it's own druntime
>> instance, they need to share one provided by the host app.
>
>
> I agree, and now we ship a Phobos DLL, resolving that issue.

Really? Where is it? (I can't see it in the distribution).


>> 7. D binaries are *gigantic*, I don't know why. D seems to link the
>> world, and I'm certain that most of the crap that's linked is never
>> referenced or executed... it just makes gigantic binaries for some
>> reason, even with line-for-line translations of C/C++ code. That's not
>> real good on games consoles where icache is king.
>
>
> Yes, that's an ongoing problem. I need to spend some time on that one.

That'll be nice. Bear in mind, I'm just giving an account here of why
we were unable to produce blog posts and hype from our experience
using D at Remedy, at least up until the time I left, I'm not sure
where they are with it now. I hope it didn't fizzle, but there were
more problems than we initially hoped for, and I think that damaged
confidence a lot.

In the short term? I struggle to imagine the ddmd transition taking
less than 3-6 months.


Personally, if I were to pin a #1 ticket related to my 2015 workload,
it would be Emscripten support, and LDC as default D compiler ;)


>> I tried to use D in my current workplace on 3 occasions now, and ran
>> into a different set of problems.
>> In the first attempt I tried to write a simple vibe.d webserver app,
>> but it kept crashing randomly when I was using the API's exactly as
>> intended (apparently a client was posting `/n` instead of `/r/n`, or
>> maybe the inverse of that, and it didn't like it).
>
>
> Were these bugs reported?

Yup, and some of them fixed. Other's remain because I couldn't create
a reduced test environment; async communications using websockets in
this case.
At the time I was actually heckled by people here telling me I was
stupid to try and use vibe.d commercially.
I was fairly surprised, it seems like one of the D community's biggest
success stories, and people were (almost violently) telling me I'm
stupid to attempt to use it in production.


>> In my second attempt, progress was interrupted by some ICE's, linking
>> with some C++ code caused various problems.
>
>
> ICE's in what? What problems?

LDC. I think it's the most high-priority tool in the ecosystem, and it
doesn't have anywhere near enough manpower. I am 100% in favour of the
many calls for LDC as front-line compiler.

Might I suggest - and I'm sure you won't like this, but I think it
would do worlds of good; your experience implementing MS debuginfo in
DMD is invaluable and fairly unique. If you could contribute that
debuginfo support to LLVM, it would make a world of difference.


>> A final recent attempt was basically a non-starter on account of no Qt
>> bindings that work &&/|| are up to date.
>
>
> Bindings aren't too hard to update.

Qt is not a small library. Binding Qt appears to be a much, much
bigger task than my task. I'd need to take it as an ongoing hobby
project, but I'm overrun with those already >_<


>> I'm getting pretty close to calling it a day. Is 6 years enough time
>> to repeatedly prove to myself that I'm chasing a fantasy?
>> People keep talking about Rust, I'm thinking of giving it a shot.
>>
>> I think D needs a budget, and paid programmers employed to do boring
>> shit that makes the whole thing more stable and robust. Mozilla seems
>> to have this(?)
>> D seems to consistently offer 95% solutions to problems. 95% is not a
>> solution, it's just another problem.
>>
>> Oops, I ranted. Sorry! :)
>
>
> I think what stalled Remedy's use of D is you left Remedy. Note that while
> you were there I went to Finland largely at my own expense to help make it
> work for Remedy. After you left, there wasn't much interest from Remedy.

You're probably right to an extent, I was really pushing for it and I
had the latitude to do so at the time, but interest was actually very
high. Most commercial dev's I've worked with tend not to engage in the
open-source space though, so maybe it just wasn't part of their habit
to involve in communities like this one. I really do appreciate the
effort you made. I hope it was valuable development regardless, I
think a lot of important things came form it.

The reality is that it did take a lot longer to get to a workable
state than we had initially budgeted though. Off the back of that, I
think there was a lot of confidence lost, and we also entering a phase
of crunch.
The gravity of crunch in the video games industry can't be
understated; once the crunch coding begins, there is very little time
in anyone's life to get some new experimental tech across the line. We
reached the point where we were able to implement systems in D and it
was looking great, but by that time, none of the systems we planned to
implement in D were written in D because their deadlines had already
passed, so they fell back to C instead, and momentum was lost.


> I note that I rarely know just what is blocking you. For example, you say
> above you got some ICEs. What ICEs? We cannot fix bugs we don't know about.

LDC bugs, I reported them all. I couldn't sufficiently reduce test
cases in most cases though. The latest release has some promising
looking bugs fixed and I'm keen to try it out again on that code, but
the time for that code has passed so it's just another opportunity
missed at this point. I'll try it out enthusiastically, but what are
the odds I'm blocked again a couple of hours later by some other minor
and frustrating thing?

> I would also very much appreciate if you would pitch in to help with things
> that are blocking you, like updated Qt bindings. Generally, people around
> here tend to contribute fixes for things that personally interest or block
> them. Priorities for you aren't priorities for others.

I understand, and I've come to terms with this. But the practical
effect of that is sadly just to distance me from D. And I'm honestly
not trying to be negative, just realistic.
I looked into writing Qt bindings, but it's a pretty big job, probably
much bigger than the job they are to be used for. The question I have
to ask though is, would that enable me to do my work, or would I just
run into the next problem? That was the blocker, but there may have
been endless problems after that.
I can't reasonably gamble with my deadlines in that way. It's a bit of
a chicken-and-egg problem.

Sadly, I'm working more overtime than ever this year, so I just don't
have after-hours at the moment. You might have noticed my relative
silence.
I'm still trying to polish up std.color (which was a blocker for one
of my after-hours projects) in the free time I have. It's kinda
sitting in limbo waiting on ctfe float ^^ operator, and also
reviews...


> D doesn't have a community so large that you can sit back and assume many
> others will run into the same issues you have and will fix them. You have to
> be more proactive.

I understand, but I don't have time to allocate.
The only practical suggestion that comes to mind is that D needs a
budget, and employees. People paid to take the really boring stuff
seriously, and ideally prioritise paying corporate users, such that
they can have confidence that their issues won't be show-stoppers.
That probably wouldn't work well in practise though. While the number
of bugs and issues in DMD are steadily declining, I'm finding the
show-stoppers are just moving elsewhere.
LDC has ICE's in the quantity that DMD used to have, Android and iOS
support (ie, most computers in the world) are moving along rather
slowly; they are single-man teams and don't have core support.
Emscripten, NaCl, etc aren't supported.
Almost all my recent work can't ignore these platforms. Bugs relating
to external blockers like these don't belong in the DMD database.

In fact, we've been discussing for a few months that we'd have have
another very promising opportunity to use D at work in a really
appropriate context if I could rely on Android and iOS appearing
within 6-12 months or so. It would take a plug-in form like at Remedy,
which I think is a really practical transition path for a company with
engineers that need opportunities to get their feet wet, but can't
afford to bet the farm.


> Looking at open bugzilla issues you've commented on:
>
> https://issues.dlang.org/show_bug.cgi?id=5227
> https://issues.dlang.org/show_bug.cgi?id=6343
> https://issues.dlang.org/show_bug.cgi?id=8845
> https://issues.dlang.org/show_bug.cgi?id=9065
> https://issues.dlang.org/show_bug.cgi?id=9137
> https://issues.dlang.org/show_bug.cgi?id=10203
> https://issues.dlang.org/show_bug.cgi?id=10959
> https://issues.dlang.org/show_bug.cgi?id=11541
> https://issues.dlang.org/show_bug.cgi?id=12440
> https://issues.dlang.org/show_bug.cgi?id=13265
> https://issues.dlang.org/show_bug.cgi?id=14260
>
> That's it. None of them seem to be ones you brought up in your post.

This small number actually represents how far DMD has come recently. I
recognise and appreciate how far the compiler has come; I don't have
codegen or language issues very often at all these days. And likewise,
they haven't been the focus of my complaints. LDC not supporting
debuginfo for instance doesn't appear on the DMD bugzilla.

Note, x^^y is still in that list. That's blocking std.color ;)

> To get attention for issues that are blocking you you absolutely have to file
> bugzilla reports. That doesn't guarantee they will be fixed, but if you do
> not file them, it DOES guarantee that they will NOT be addressed, and you'll
> get frustrated.

I think you misunderstood the point of my rant. What I was trying to
share is past experience, not the state of the ecosystem today. It's
important to note the recurring experience where there always seems to
be some random thing that blocks me. I've approached many different
kinds of projects now, from a lot of different angles. I'm still
unable to make one stick. I'm obviously doing something wrong
repeatedly, but you can't say I haven't given it a jolly good effort!

I'm pretty consistent at logging bugs, except those that I can't
justify burning a few hundred dollars of work time to produce a
reduction that I could log are often lost (and often quite important).
I'm often inhibited by the fact that I attempt to use D at work, and
offices are stupid proprietary places so I can't share code >_<

My job and workloads have changed a lot recently, so I'm facing new problems.
But yeah, LDC and missing libraries+platforms are the source of almost
all my issues in 2015. It would be great for DMD devs to be building
LDC routinely, and this situation where GDC and LDC are a couple of
versions behind DMD has to end asap.


More information about the Digitalmars-d mailing list