[dmd-internals] 9th Sprint Planning

Fri Nov 27 19:58:49 PST 2015

Long overdue, welcome to the 9th sprint planning.

4 weeks means there is a lot to talk about, so please read the somewhat
lengthy mail and participate in the planning, it's well worth your time.

After the unplanned 2.069.1 point release we'll now start the beta for
the 2.069.2 bugfix release which is scheduled to be released on 3rd of
december.
As we don't have much regression reports the release should be very
small. We're still urgently searching for someone to include
std.experimental.allocator in phobos' windows makefiles.
https://trello.com/c/thMHmnvz/133-issue-15281-std-experimental-allocator-package-d-not-included-in-build-script

##### Post-mortem of Windows installer bug #####

During the update of the NSIS installer to support VS2015 a bug was
introduced that caused an infinite loop (it was caused by a misplaced
label). It took us a few days to notice the multiple forum comments.
It then took a few more days to get someone affected by this bug to help
us, b/c we were unable to reproduce the issue.
After quite some trial and error and spending more than 1 hour to build
a NSIS installer with logging, we finally managed to pinpoint the bug
and Rainer made a fix.

Last time I looked at the histogram, about 50% of our users download the
Window installer. Because of this severity we decided to immediately
follow up with a point release.

There are a few learning from that.

1.) We don't systematically test our installers. For the Windows
installer it would be installing w/ and w/o existing dmd installation
times w/ and w/o existing VS installation. We really need some
volunteers testing those 4 combinations.

2.) Using NSIS is a major pain. The platform hasn't been updated since 6
years (though suddenly there is a beta for NSIS 3.0).

I have to host the unzip plugin binary myself before it vanishes from
the internet, we need to use a "special" NSIS build that's able to deal
with "large" strings (more than 1KB).
https://dlang.dawg.eu/downloads/others/
http://nsis.sourceforge.net/Nsisunz_plug-in
http://nsis.sourceforge.net/Special_Builds#Large_strings

But the worst thing is that the installer script is written in a sort of
macro assembler, which is the actual cause for the bug.
https://github.com/D-Programming-Language/installer/blob/cc4d45a4cdde945ea2cd4eeb3dd05b4289f786ee/windows/d2-installer.nsi

3.) It would be better to integrate with Windows Installer and use a
.msi installer.

https://en.wikipedia.org/wiki/Windows_Installer

We got a friendly recommendation to use InnoSetup for nicer .msi
installers. I'm not yet willing to spend the effort to rewrite the
installer, but on the next occasion (bug) we should do it.

##### Review of last sprint #####

I haven't followed the upstream development too closely, so maybe
someone else can fill in more details.
I just want to use the occasion to highlight how spending some thought
and fixing an issue properly, instead of adding yet another hack on top
of an already messy implementation, can profoundly improve our code base
and subsequently allow to more easily implement other improvements.

https://github.com/D-Programming-Language/druntime/pull/1419#issuecomment-152548521
https://github.com/D-Programming-Language/druntime/pull/1421
https://github.com/D-Programming-Language/druntime/pull/1422
https://github.com/D-Programming-Language/druntime/pull/1110

I spend most of the past 4 weeks with infrastructure work.

## dlang-bot

As you may have noticed already there is now a dlang-bot on github
posting bugzilla links on PRs, so that people no longer have to do this
this manually and can spend their time on useful PR descriptions instead.

https://github.com/dlang-bot?tab=activity

This was also good for a small article (w/ a problematic code/words
ratio), that was read by quite a few people.
https://code.dawg.eu/writing-a-vibed-app.html

I'm planning on another vibe.d article about a git based wiki app I'm
already using since a few years.

## nightlies and installer scripts

I spend a lot of time to setup nightly builds. First I was using my
local dev machine with a rtc wakeup and cron job, but that turned out to
be too unreliable (kernel/package updates, S3 wakeup hang, forgot to
unlock gpg key). So I now migrated this to a dedicated server (a low end
box that's 4x slower than my dev machine) and hope to stabilize and
publish everything over the next few days.
You can find a beta of the nightlies service here.

https://builds.dawg.eu/

If you want to install a nightly you might try out our new install
script (and yes that safe to pipe to bash [¹]).

curl -fsSL https://builds.dawg.eu/install.sh | bash -s dmd-nightly

If you don't like that, do something else, e.g.

curl -fsSLO https://builds.dawg.eu/install.sh
curl -fsSLO https://builds.dawg.eu/install.sh.sig
gpg --verify install.sh.sig
vim install.sh
chmod +x install.sh
./install.sh dmd-nightly

or just download the latest nightly directly.

https://builds.dawg.eu/dmd-nightly/dmd.master.linux.tar.xz

The installer script with create a ~/dlang folder, installs itself, can
install almost any version of dmd/gdc/ldc, is based on the travis-ci and
heroku script and should be pretty self-explanatory.
If you find something non-obvious, please help to improve it [²].

I hope to find some time soon for a Windows powershell version,

(new-object
Net.WebClient).DownloadString("https://builds.dawg.eu/install.ps1") | iex

though I'd prefer if someone else would be up for that task (I never use
Windows myself).

[¹]:
https://github.com/MartinNowak/installer/commit/2d7c07066cfee1a610dda2026e4d4c4e7b41fa5f
[²]: https://github.com/D-Programming-Language/installer/pull/162

I also intend to expand on that dmd-nightly semantic and add dmd-beta
(using http://ftp.digitalmars.com/LATEST_BETA).

Once the nightlies are stable enough they should become a dlang.org
subdomain, e.g. nighlies.dlang.org, and get a place on the download
page. Together with the corresponding one-step installer script lines.

curl -fsSL https://dlang.org/install.sh | bash -s dmd
curl -fsSL https://dlang.org/install.sh | bash -s dmd-beta
curl -fsSL https://dlang.org/install.sh | bash -s dmd-nightly

## Progress on OSX shared libraries

I don't want to announce anything unfinished, but with Nicolas
Jinchereau (bitwise) we finally seem to have found someone capable of
putting together an OSX shared library implementation for dmd. Thanks
for your great work so far.

##### Planning of next spring #####

I'll build 2.069.2-b1 tomorrow. The project tester [¹] didn't show any
new issue and I'm not aware of any 2.069.x regressions being reported.
Please test the beta asap, so we can ship 2.069.2 on time (2015-12-03).

Besides the nightlies, I only plan to finish the GC spinlock
implementation [²] and ignoring GC.free in finalizers [³],
as I'll be on vacation from 5th till 19th of december.

Maybe there is some extra time to track down the remaining slist_reset
bug for OMF [⁴] (help appreciated) or finish the smart_ref library, but
it's not very likely.

[¹]: https://github.com/MartinNowak/project_tester
[²]: https://trello.com/c/Xp44NwHG/87-ttas-spinlock-for-gc
[³]:
https://trello.com/c/1UXjGQBf/135-issue-15353-std-experimental-allocator-cannot-free-memory-in-its-destructor-if-the-gc-is-an-ancestor
[⁴]: https://github.com/D-Programming-Language/dmd/pull/5109

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.puremagic.com/pipermail/dmd-internals/attachments/20151128/b4c2b4cc/attachment.sig>