D for Game Development

[Manu via Digitalmars-d]

On 10 August 2015 at 06:51, Walter Bright via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
> On 8/9/2015 4:38 AM, Manu via Digitalmars-d wrote:
>>
>> On 9 August 2015 at 15:31, Walter Bright via Digitalmars-d
>>>
>>> But waiting for someone else to discover the same thing on some other
>>> piece
>>> of code means you'll be waiting a long time.
>>
>> I understand, but that's not sustainable.
>> We can't be in a situation where I'm at work with a deadline, and I
>> need to invoke the community to address a codegen issue that's
>> blocking a release. Unexpected surprises like that with unknown/long
>> turn-around times aren't a practical commitment.
>
>
> It will NEVER get fixed if you don't file bug reports, and you'll continue
> to be frustrated (and me too, because there's NOTHING I can do to improve
> things based on what you post here).

I think you misunderstood my point again. I'm trying to say that it's
precarious to make a commitment to something you know (or suspect) has
unknown issues. The practical outcome is that there's a strong
pressure against making such a commitment (ie, not use D), and then
certainly no bugs will be found or reported.

The thing is, whatever bugs may or may not be discovered are almost
certainly already 'fixed' in LLVM/GCC. There's no work to do there,
those backends have legions of developers working on them every day.
There is literally nothing DMD can ever to do catch up. It may be sad
to admit, but it's just how it is. Maintaining and fixing the DMD
codegen just takes time away from other things (I understand you argue
that it's very little time in practical terms, but that's not really a
great argument; it just admits that DMD is not progressing), and it
has lead to a fragmented experience, which is the biggest practical
issue I've been finding with D in recent times.
DMD as a frontend to LLVM seems like the most practical and reliable
option as the default offering.

You did a lot of work to make debugging in DMD work well, and the work
really paid off! DMD is the only windows compiler that produces a
workable debugging experience.
The consequence of that though, is to have created a situation where
people necessarily rely on a tooling situation where they use 2
different compilers for debug and release, and because the other
compilers are always a few versions out of date, you're behind on all
the latest developments.
If that effort to reverse engineer Microsoft's debuginfo were
contributed to LLVM, almost every native developer, D or any language
otherwise, would all be better off.

So the point is, many of my modern D issues aren't issues that I can
bug. I can't log on bugzilla: "using DMD for debug and LDC for release
builds sucks; fix debuginfo in LLVM". That's obviously outside the
scope of DMD, but I suggest that maybe it shouldn't be?
DMD has become fairly reliable for me, I don't fight it much anymore
(or maybe I'm just much better at knowing what works well?), but the
larger ecosystem seems to be the biggest cause of friction now. These
aren't problems that are easily directed to a particular bug tracker,
they represent a conjunction of tooling, and maybe strategies need to
be considered where core developers are aware of these issues.

Leveraging the world of LLVM tooling is the most practical and
bang-for-buck solution I can imagine. The other advantage of core dev
using an LLVM backend, would be that port/architecture related bugs
would become the business of core development. CI would necessarily
have to build for many architectures, and PR's blocked until they work
on ARM/iOS/etc.


>> Last time I was concerned with this, x87 was destroying DMD
>> performance. Floats were x87, moved to XMM (according to x64 ABI) when
>> passed to functions, then moved back into x87 on the other side when
>> they're operated on.
>> Float code was overwhelmed with shuffling values between registers.
>> Basically all hot code is float code in my line of work.
>
>
> All I really need is a bugzilla issue with just ONE example of needlessly
> using the x87 where a SIMD instruction would be better.

Like I said, it's not a problem I have today. I was answering a
question and discussing historical problems.


>> I think Ethan was in discussions with you about this some time back?
>> It was a problem at the time I left Remedy.
>
>
> As I recall, I heard from Ethan just once since you left.

That's a bummer. Sorry to hear that. I should catch up with those guys
and see where they're at.


> In contrast, I hear regularly from Sociomantic when they've got an issue, and I'm able to
> consequently help them be successful with D.

Thing is, most of my 2015 blockers are outside the scope of DMD unless
DMD decides to adopt LLVM as it's backend.
DMD is in pretty good shape these days, it's just a shame it can't be
used to compile software that runs on modern computers (arm, iOS,
Android, web).

We all need Android, iOS, Emscripten. We need equal confidence the
ports work well as that DMD is released to a high quality. I think
they should be built by the DMD core CI, all platforms tested for each
release, cross-compilers released simultaneously, and PR's rejected
when they fail on other arch/OS's.
Unless you feel these are DMD core issues, then like I say above, I
have nothing to bug except for minor issues like x^^y, which are
logged, and that large static struct issue, which you fixed quickly.

I'm really hoping that the DDMD transition will give an opportunity
for all compilers to truly share the frontend code, so they can all
track the same frontend revision at the same time, and PR's can fail
CI out of the gate if they break other platforms/architectures.
I've posted on so many occasions about how important the mobile ports
are; immeasurably more so than their developer representation (1 guy
each).
If you feel I haven't communicated this with you effectively, I'm
sorry. I personally feel I sound like a broken record on this, but I
also haven't felt like you're the right guy to hassle personally about
these things. I do however think it would be extremely valuable for
you to take a strong interest in LDC and it's health. Realise, when
LDC doesn't work, it's letting DMD and the whole D experience down
too.

(*note: as in prior posts, I don't intend to down-play GDC; it's also
very important. It's just that LDC has better platform coverage)


>> Also, I think one of the big issues is the optimiser in general. The
>> inliner's just not fantastic, and D depends on it perhaps even more
>> than C++, especially when using the modern lazy code style.
>> I'm sure I can hand-assist DMD to produce the assembly I expect for
>> some hot function with some work, but traditional hand-assisted code
>> generation all falls apart when you start stacking lazy-ranges and
>> generally using the modern D style.
>> DMD's backend has almost no development, it's advancing at near-zero
>> velocity compared to LLVM. We need to get out of the situation where
>> LDC and GDC are perpetually 1-2 versions behind.
>>
>> Incidentally, In my recent (dead-end) experience with LDC, I get
>> endless ICE's whenever I try to do anything mildly complex. When you
>> have your Debug and Release build straddling language versions and
>> compilers with different ICE characteristics, it's just not a
>> practical workflow. Only green-fields projects or the simplest of
>> simple apps can hope not to trigger bugs constantly in that
>> environment.
>
>
> I can't help you with LDC ICE's, but are you filing bug reports for them?
> Again, if you don't file bug reports, NOTHING WILL GET FIXED. You will be
> frustrated and everyone else will be, too.

I'm not sure what I've said to give you such a strong impression I
don't file bugs. I almost always do unless they would take me many
hours of paid work time to reduce, that's the only situation in which
they slip.

I feel like you've taken my post way out of context, or really
personally somehow. I don't mean that. I was just giving an account of
historic happenings in response to a question.


>>>> 2. LDC has a lot more bugs than DMD (which still has quite a few
>>>> bugs), and the Windows build is new and even more immature still.
>>>> 3. LDC has no debuginfo. **biggest practical issue by far!**
>>>> 4. LDC/GDC are always a few versions behind DMD. This creates an
>>>> awkward/almost-impossible situation when you rely on DMD to build
>>>> debug code, and LDC to build releasable code. Depending on 2 flaky
>>>> compilers is even less fun than one.
>>>> 5. DLL's were really flakey at the time, I'm not sure how they are
>>>> now, but I'm concerned they may not be much better.
>>>> 6. Shared druntime; we were loading D code from DLL's, lots of them.
>>>> It's not really reasonable for each DLL to have it's own druntime
>>>> instance, they need to share one provided by the host app.
>>>
>>>
>>>
>>> I agree, and now we ship a Phobos DLL, resolving that issue.
>>
>>
>> Really? Where is it? (I can't see it in the distribution).
>
>
> Should be in the bin directory.

Manu at MANU-PC /c/dev/D/dmd2/windows/bin
$ ls *.dll
dm.dll  eecxxx86.dll  emx86.dll  libcurl.dll  mspdb41.dll  shcv.dll  tlloc.dll


Anyway, it's great to hear it's come good. I don't need it right now,
but it'll be useful again in the future I'm certain.


>> At the time I was actually heckled by people here telling me I was
>> stupid to try and use vibe.d commercially.
>> I was fairly surprised, it seems like one of the D community's biggest
>> success stories, and people were (almost violently) telling me I'm
>> stupid to attempt to use it in production.
>
>
> I didn't see such comments, but there'll always be naysayers.
>
>
>> Might I suggest - and I'm sure you won't like this, but I think it
>> would do worlds of good; your experience implementing MS debuginfo in
>> DMD is invaluable and fairly unique. If you could contribute that
>> debuginfo support to LLVM, it would make a world of difference.
>
>
> All I know is actually in the DMD source code.

Sure, but that's probably not the best place for it. You did the hard
yards, but without putting it in LLVM it's utility is significantly
reduced.
I've actually had quite a few conversations with LLVM dev's about
this, and one of the key excuses for not supporting MS debuginfo was
the classic "it's not documented". I promptly pointed them at your
work every single time it came up, but they ignored or dismissed it
every time. Dev's are funny like that.


>>> Bindings aren't too hard to update.
>>
>> Qt is not a small library. Binding Qt appears to be a much, much
>> bigger task than my task. I'd need to take it as an ongoing hobby
>> project, but I'm overrun with those already >_<
>
>
> You wouldn't need to do the whole thing, just an update to the existing one,
> and just enough to get your project moving. Does your project use every Qt
> declaration? :-)

Qt5 is a significant development away from Qt4, and I didn't find any
existing Qt binding that touches Qt5.

The hard part is reproducing moc, and uic. If you haven't had contact
with Qt, you'll need to read up on what they are for.
I'm confident a quality D binding for Qt would do away with their
suite of annoying auxiliary tools; we could do their job with compile
time magic and mixins, but reproducing those tools in CTFE represents
a lot of work. You can't do anything with Qt until they work.

Auxiliary build tools (like moc/uic) are an annoying facet of any
build system, and on doing away with those I think D would be a
fantastic language for using Qt, but it's a lot of work.


>> Note, x^^y is still in that list. That's blocking std.color ;)
>
>
> It has trivial workarounds.

Kinda.

void f(T)()
{
  static immutable float someData[256] = ctfeGenTable!T(); // <- uses x^^y

  ...use `someData`...
}

Assume there are many possible T's.
Lazy initialisation of tables like this feels pretty rough. Runtime-if
to populate someData doesn't feel great, particularly because in my
particular case of interest, these are very high frequency functions.
I don't want to commit such a hack to phobos.


>> I'm obviously doing something wrong
>> repeatedly, but you can't say I haven't given it a jolly good effort!
>
>
> I do appreciate that, and your efforts.
>
>
>> I'm pretty consistent at logging bugs, except those that I can't
>> justify burning a few hundred dollars of work time to produce a
>> reduction that I could log are often lost (and often quite important).
>> I'm often inhibited by the fact that I attempt to use D at work, and
>> offices are stupid proprietary places so I can't share code >_<
>
>
> If your bug reports require an NDA, I'd be happy to sign one and we can
> handle the bug report privately. In any case, by the time a bug report is
> reduced to one suitable for filing, I've never seen one that had any
> information in it that was of proprietary value.

It's about the time available to spend on complicated reductions while
I'm on the clock. This is the only reason in my case that bugs don't
lead to bug reports.
Considering your comparison to Sociomantic before; I suspect they have
more than one guy in the building that would actually do this stuff.


> At least for targets that DMD supports, I recommend sticking with DMD if you
> are blocked by LDC. At least get the project working and stable before
> worrying with LDC specific problems.

But a project is a non-starter if there's no evidence that LDC will
work according to our release schedule. You wouldn't start a project
and get it working with the expectation that you have to rewrite it in
C because the compiler doesn't support your target platforms (and
never claimed that it would). It's a super-hard-no-room-for-argument
blocker.
I can say with confidence, that if I have any reason to believe LDC
would work well on Android, iOS, Emscripten, I would have definitely
taken the time to write Qt support... I'm suffering C++ harder than
I've ever suffered before at the moment! >_< .. My colleagues are
actually really keen to have me try out D at work, if I could just
demonstrate that it met our starting criteria.

So, back to the beginning, I encourage you seriously consider LDC as
frontline compiler. I can't think of anything that would make more of
a positive impact to the D ecosystem than if the core dev's are
building LDC, core CI tests and PR's must pass on all
platforms/arch's, cross-compilers get simultaneous release with DMD,
and wider platform/arch support became the business of core dev's (and
the DMD bug tracker).
DMD has an accepted responsibility to x86, x64, and win, linux, bsd,
osx... why not widen that to arm, android, ios at least? These are the
most common computers in the world after all.

I really hope this is a top-priority goal for the switch to DDMD.