DConf 2013 Closing Keynote: Quo Vadis by Andrei Alexandrescu

[Leandro Lucarella]

Joakim, el 26 de June a las 17:52 me escribiste:
> On Wednesday, 26 June 2013 at 11:08:17 UTC, Leandro Lucarella wrote:
> >Joakim, el 25 de June a las 23:37 me escribiste:
> >>I don't know the views of the key contributors, but I wonder if
> >>they
> >>would have such a knee-jerk reaction against any paid/closed
> >>work.
> >
> >Against being paid no, against being closed YES. Please don't even
> >think
> >about it. It was a hell of a ride trying to make D more open to
> >step back now.
> I suggest you read my original post more carefully.  I have not
> suggested closing up the entire D toolchain, as you seem to imply.

Well, I'm not. I'm sticking with what you said.

> I have suggested working on optimization patches in a closed-source
> manner and providing two versions of the D compiler: one that is
> faster, closed, and paid, with these optimization patches, another
> that is slower, open, and free, without the optimization patches.

I know, and that's what my e-mail was all about. I don't know why you
got another impression. I even end the e-mail saying is a very bad
business model too to just offer a paid better optimizer.

> >What we need is companies paying to people to improve the
> >compiler and toolchain. This is slowly starting to happen, in
> >Sociomantic we are already 2 people dedicating some time to
> >improve D as
> >part of our job (Don and me).
> Thanks for the work that you and Don have done with Sociomantic.
> Why do you think more companies don't do this?  My point is that if

Because D is a new language and isn't as polished as other programming
languages. I think Sociomantic was a bit crazy to adopt it so early
really (my personal opinion). But it worked well (we had to do quite
a lot extra efforts but I guess the time it saves in the daily usage
paid for it).

> there were money coming in from a paid compiler, Walter could fund
> even more such work.

Well, I think with a paid compiler you remove one of the main reasons
why early adopters can be tempted to use D, because is free. What I'm
sure is Sociomantic wouldn't pick D if they had to paid at that time,
because it was a statup and startup usually don't have much money at
first.

> >We need more of this, and to get this, we need companies to start
> >using D, and to get this, we need professionalism (I agree 100% with
> >Andrei on this one). Is a bootstrap effort, and is not like
> >volunteers need more time to be professional, is just that you have
> >to want to make the jump.
> I think this ignores the decades-long history we have with open
> source software by now.  It is not merely "wanting to make the
> jump," most volunteers simply do not want to do painful tasks like
> writing documentation or cannot put as much time into development
> when no money is coming in.  Simply saying "We have to try harder to
> be professional" seems naive to me.

Well, I guess we have very different views about the decades-long
history of open source software, because I know tons of examples of
applications being free, without "commercial implementations" or "paid
modules" and very few with a more commercial model. Even more, the few
examples I know of "paid modules" are quite recent, not decades-old.

> >I think is way better to do less stuff but with higher quality,
> >nobody is asking people for more time, is just changing the focus
> >a bit, at least for some time. Again, this is only bootstrapping, and
> >is always hard and painful. We need to make the jump to make
> >companies comfortable using D, then things will start rolling by
> >themselves.
> If I understand your story right, the volunteers need to put a lot
> of effort into "bootstrapping" the project to be more professional,
> companies will see this and jump in, then they fund development from
> then on out?  It's possible, but is there any example you have in
> mind?  The languages that go this completely FOSS route tend not to
> have as much adoption as those with closed implementations, like
> C++.

Are you kidding me? Python, Ruby, PHP, Perl. Do I have to say more than
that?  Do you really think C++ took off because there are commercial
implementations? Do you think being a standardized language didn't help?
Do you think the fact that there was a free implementation around that
it supported virtually any existing platform didn't help? Do you think
the fact was it was (almost) compatible with C (which was born freeish,
since back then software was freely shared between universities) didn't
help?

> >First of all, your examples are completely wrong. The projects you
> >are mentioning are 100% free, with no closed components (except for
> >components done by third-party).
> You are misstating what I said: I said "commercial," not "closed,"

You said close. Not just in the previous e-mail, but you just repeated
it in this one:
> I have suggested working on optimization patches in a CLOSED-SOURCE
> manner and providing two versions of the D compiler: one that is
> faster, CLOSED, and paid, with these optimization patches, another
> that is slower, open, and free, without the optimization patches.

I think you are misstating yourself ;)

> >Your examples are just reinforcing what
> >I say above. Linux is completely GPL, so it's not even only open
> >source.  Is Free Software, meaning the license if more restrictive
> >than, for example, phobos. This means is harder to adopt by companies
> >and you can't possibly change it in a closed way if you want to
> >distribute a binary.
> And yet the linux kernel ships with many binary blobs, almost all
> the time.  I don't know how they legally do it, considering the GPL,
> yet it is much more common to run a kernel with binary blobs than a
> purely FOSS version.  The vast majority of linux installs are due to

That's because the binary blobs are not part of the kernel, is firmware
for certain (crappy) hardware that needs the firmware to be loaded to
work. For convenience, the kernel distribute it so this (crappy)
hardware can work out of the box. But this is produced by third-parties,
not kernel developers. This is a completely different example and is
giving NO MONEY AT ALL to linux development.

> Android and every single one has significant binary blobs and
> closed-source modifications to the Android source, which is allowed
> since most of Android is under the more liberal Apache license, with
> only the linux kernel under the GPL.

OK, so by Android you mean the entire platform, not the kernel.

> Again, I don't know how they get away with all the binary drivers in
> the kernel, perhaps that is a grey area with the GPL.  For example,
> even the most open source Android devices, the Nexus devices sold
> directly by Google and running stock Android, have many binary
> blobs:
> 
> https://developers.google.com/android/nexus/drivers

Again, but this gives is not a model adopted to make money to improve
Android, this is just drivers third-party do so android can run in their
devices.

See the difference? Is not like Android developers said: "we need money
to be able to code Android more professionally, so we are going to write
closed source driver and sell them!". The companies making the closed
source drivers are the hardware retailers.

> Other than Android, linux is really only popular on servers, where
> you can "change it in a closed way" because you are not
> "distributing a binary."  Google takes advantage of this to run
> linux on a million servers powering their search engine, but does
> not release the proprietary patches for their linux kernel.

They do contribute with the mainline kernel though (not as much as they
should). But then I think you are digressing a little. You were saying
D should offer an improved, closed source, paid optimizer to raise money
to improve the quality of the compiler. How this Google example applies
to this?

> So if one looks at linux in any detail, hybrid models are more the
> norm than the exception, even with the GPL. :)

Google taking advantage of the flaws of GPL 2.0 doesn't mean the Linux
kernel is implemented an hybrid closed/open source model. What you are
saying doesn't make any sense, you are mixing everything together.
Seriously.

The Linux kernel have NO hybrid model, is 100% opensource. Then, there
are companies using it, and when they are forced, or it is convenient
for them, they contribute to Linux. And this is what I'm suggesting is
best for D too. We have to either force or make it convenient to
companies to contribute.

> >Same for C++, which is not a project, is a standards, but the most
> >successful and widespread compiler, GCC, not only is free, is the
> >battle horse of free software, of the GNU project and created by the
> >most extremist free software advocate ever.
> D is not just a project but a standard also.

No. A standard is something that was standardized by a standard
committee which, ideally, have some credits to do so. C++ is
standardized by ISO. I guess Walter and Andrei can give you more
details, since I think they both were involved in the standardization of
C++. D is a language specification and a reference implementation. Which
is not the same.

> I wouldn't say gcc is the "most successful and widespread compiler,"
> that's probably Microsoft's compiler, since Windows market share is
> still much more than linux.

Are you counting devices or just personal computing? You know all
Android phones don't use Microsoft compiler, you know all your gadgets
don't user Microsoft compiler, you know your car software was not
compiled with Microsoft compiler, same your TV and so many other stuff.

> But yes, gcc is a very popular open-source implementation: I didn't
> say there wouldn't be an open-source D compiler also.  But I don't
> think C++ would be where it is today if only open-source
> implementations like gcc supported it, which is the case today with D.

And how do you explain the success of languages that don't have (at
least major) commercial implementations like Perl, PHP, Python or Ruby?
Or even Go? I think Go is much more suitable for companies right now
than D, because in Go they put a lot of emphasis in the toolchain and
polishing the tools. You might say "Go was done by Google" and is true.
But there are NO COMMERCIAL implementations. Only free. And they are
tempting for companies because Go development is professional (thanks to
Google support of course). The thing is, again, we need companies
support. Not commercial implementations. Well, commercial
implementations are welcome though, what we certainly don't need is the
reference implementation being (partly) closed.

> >Android might be the only valid case (but I'm not really familiar
> >with Android model), but the kernel, since is based on Linux, has to
> >have the source code when released. Maybe the drivers are closed
> >source.
> As I said earlier, most devices' drivers are almost always closed
> and the non-GPL parts of Android, which are the majority, are
> usually customized and the source is usually not released, because
> most of Android is Apache-licensed.  I think this closed option is a
> key reason for the success of Android.  Hell, the hardware vendors
> would never have adopted Android if not for this, as Google well
> knew.

Again, I think this is a bad example because those closed sourced
elements are not making money to support Android developers. Is what
hardware retailers need to do to make their phones work with Android and
they don't want to release the code of their drivers.

> >You are missing more closely related projects, like Python, Haskel,
> >Ruby, Perl, and probably 90% of the newish programming languages,
> >which are all 100% open source. And very successful I might say. The
> >key is always breaking into the corporate ground and make those
> >corporations contribute.
> I believe all of these projects have commercial implementations,
> with the possible exception of Haskell.

Name them please. I would like to know them.

> Still, all of them combined have much less market share than C++

[citation needed]

Not that they are extremely reliable, but sites measuring language
popularity say Python+PHP+Perl+Ruby have more share than C++.

TIOBE sais:
	P+P+P+R=13.922 vs C++=8.819
The Transparent Language Popularity Index sais:
	P+P+P=R=11.354 vs C++=7.544

Also C++ have 10 years advantage, and 10 REALLY good years, where there
were almost no alternatives, so I'm guessing there is a lot of legacy
C++ code. It would be good to have numbers on the popularity of
languages for NEW projects. The 4 most popular languages are old
C/C-derived. Java is the only exception, but it had a massive marketing
campaign among corporations, and even when now is free, I think is, with
C#, the only languages that were born as one-only closed source
implementation.

> possibly because they use the weaker consulting/support commercial
> model most of the time.  One of the main reasons C++ is much more
> popular is that it has very high-performance closed implementations,
> do you disagree?

Yes. I think Java is the example you have in mind, not C++. For the
reasons I explain above. I think C++ was popular because it had
a high-performance open source implementation. 30 years ago.

> I'm suggesting D will need something similar to get as popular.

And I'm trying to prove you are wrong, that D needs to be popular is to
get professional without sacrificing openness.

Is because this model is dying that one of the most popular languages
switched from a closed source model to an open source one (Java).

> >There are valid examples of project using hybrid models but they are
> >usually software as a service models, not very applicable to
> >a compiler/language, like Wordpress, or other web applications.
> >Other valid examples are MySQL, or QT I think used an hybrid model at
> >least once. Lots of them died and were resurrected as 100% free
> >projects, like StarOffice -> OpenOffice -> LibreOffice.
> There are all kinds of hybrid models out there, some would work for
> compilers also.  I think it's instructive that you are listing some
> of the largest and most successful, mostly-OSS projects in this
> list. :)

As failed or counter examples :)

> >And finally making the *optimizer* (or some optimizations) closed
> >will be hardly a good business, being that there are 2 other backends
> >out there that usually kicks DMD backend ass already, so people
> >needing more speed will probably just switch to gdc or ldc.
> Let me turn this argument around on you: if there is always
> competition from ldc and gdc, why are you so scared of another
> option of a slightly-closed, paid compiler?

I'm not scared.  I'm against it for the reference implementation that
went through a lot of pain to become more open. DMD started as closed
source. There is a big difference. I'll be more than happy to receive
a closed source commercial implementation that is not based on open
source contributed by the community, which is what you suggested really.

> If it's not "a good business," it will fail and go away.  I think it
> would be very successful.

Start your own company and make a D compiler then! I wish you very good
luck! :)

> >As in breaking into the commercial world? Then agreed. If you imply
> >commercial == closing some parts of the source, then I think you are
> >WAY OFF.
> OK, so it looks like you are fine with commercial models that keep
> all the source open, but not with those that close _any_ of the
> source.

Again, I think you are mixing a lot of stuff together. If you speak
about a random company, I'm fine with them doing whatever they like,
free, closed, open, whatever.

If we are talking about the reference implementation for D that now is
mostly-opensource and free software, yes, I'm against closing sources
and no, I'm not against commercial models and earning money to pay to
contributors.

> The problem is that your favored consulting or support models are
> much weaker business models than a product model, which is much of
> the reason why Microsoft still makes almost two orders of magnitude
> more revenue with their software products than Red Hat makes with
> their consulting/support model.

So now we want D to be an enterprise which goal is to make as much money
as possible? Yes, I favor slightly less lucrative models in favor of
having an open source reference implementation.

> I am suggesting a unique hybrid product model because I think it
> will bring in the most money for the least discomfort.  That ratio
> is one that D developers often talk about optimizing in technical
> terms, I'm suggesting the same in business terms. :)

Well, we disagree for both ethical and business reasons. Your only
example of a commercial compiler making money is Microsoft. You know, we
are not Microsoft, we don't have the market share Microsoft have and we
can't impose our products like they can. I think selling a slightly more
optimized DMD will be a huge failure for the reasons I already said.
I think it would be even a better model to offer a more polished
toolchain instead of a slightly faster compiler. I don't think speed is
an issue for anyone right now.

Maybe Walter can have a meaningful opinion about this, being he has been
selling compiler for quite some time now.

> >>It is amazing how far D has gotten with no business model: money
> >>certainly isn't everything.  But it is probably impossible to get to
> >>a million users or offer professionalism without commercial
> >>implementations.
> >
> >Yeah, right, probably Python and Ruby have only 5k users...
> >
> >This argument is BS.
> First off, they both have commercial implementations.  Second, they

I've been looking for commercial implementations of Ruby and Python.
Hard. I only found one for Ruby, RubyMotion, which is only for iOS,
which seems to be unsupported by the official Ruby interpreter (or any
other free implementation). So is not really relevant in this case.

I couldn't find any commercial python implementation.

If you have some, please share, but the fact that I couldn't find any
suggests if they exist, they are not very popular, and thus not really
helping Python or Ruby adoption.

> still only have a small fraction of the share as C++: part of this
> is probably because they don't have as many closed, performant
> implementations as C++ does.

Again I think your reasoning behind C++ is more popular, C++ have closed
implementations -> C++ is more popular because it has closed
implementations, is way too simplistic (and wrong). I'm also not
convinced the closed implementations are more "performant" than the open
ones. MS Visual C++, done by the company that makes loads of money,
seems to be crap from the benchmarks I found:
http://www.g-truc.net/post-0372.html

ICC is supposed to be one of the fastest compilers. If you look at their
own benchmarks, they say they are better than GCC, at least for some
integer and floating point benchmarks), but I don't trust people doing
their own benchmarks very much:
http://software.intel.com/en-us/intel-composer-xe (go to benchmark tab)

A question in stackoverflow seems to indicate that the different isn't
that clear:
http://stackoverflow.com/questions/1733627/anyone-here-has-benchmarked-intel-c-compiler-and-gcc

Also there is this presentation from MySQL comparing ICC and GCC:
http://www.mysqlperformanceblog.com/files/presentations/LinuxWorld2004-Intel.pdf
In this case they see an important improvement using ICC, but the GCC
version used is quite old, I wonder how they compare now.

Is strange that there are not really many benchmarks comparing compiler
out there. Another indication that speed in C++ is not much of a selling
point for compilers. And from what I could find out is not clear there
is a huge gain in performance when using closed source compilers. With
Visual C++ is even clearly worse, and according to you that's the most
widespread C++ compiler.

> I realize this is a religious issue for some people and they cannot
> be convinced.  In a complex, emerging field like this, it is easy to
> claim that if OSS projects just try harder, they can succeed.  But
> after two decades, it has never happened, without stepping back and
> employing a hybrid model.
> 
> I have examined the evidence and presented arguments for those who
> are willing to listen, as I'm just about pragmatically using
> whatever model works best.  I think recent history has shown that
> hybrid models work very well, possibly the best. :)

Is true, there are people that base his view on religious believes. Here
I presented you lots of proof, with numbers and examples to justify what
I say. You, on the contrary, just claimed unfunded statements. There is
also our own experience. When DMD was closed source it received almost
0 contributions, of course. Each and every step taken to make the
compiler more open shielded more and more contributions. You want more
proof and more numbers, you can take a look at this:
http://www.llucax.com.ar/blog/blog/post/6cac01e1

This is even pre-git, with git the number of contributions AND THE
COMPILER quality increased exponentially. Unfortunately I don't have any
numbers on D adoption.

I also explained why I think your particular hybrid model is bad
business. D's adoption and improvement advanced slow because it WAS
closed source, it improved a lot with openness.

Because all of this, I find it very hard to believe the model you are
proposing would be beneficial for D, from a reason stand point there is
no evidence that could happen.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
PITUFO ENRIQUE ATEMORIZA CATAMARCA, AMPLIAREMOS
	-- Crónica TV