[OT] compiler optimisations

[Laeeth Isharc via Digitalmars-d]

On Saturday, 25 April 2015 at 22:05:05 UTC, Ola Fosheim Grøstad 
wrote:
> On Saturday, 25 April 2015 at 14:48:41 UTC, Laeeth Isharc wrote:
>>> I find it worrying that the evangelical D users are 
>>> perceiving D as a compiled scripting language and claim it is 
>>> similar to Python... D semantics are not at all like Python. 
>>> That can't win.
>>
>> Why does it worry you?  What bad things will happen?
>
> Bad things that could happen is that D never can be like Python 
> and if you try to make it such you no longer have a system 
> programming contender.

So because some people have found it useful in that domain and 
have shared their positive feelings, there is a risk that this 
hijacks the direction of the language away from what would 
ultimately be to its greatest benefit (and perhaps to yours, 
anyway)?  "Nobody goes there anymore - that place is too popular".

Conceivable, but you can hardly control what people do with and 
say about their use of a programming language, even of a closed 
source commercial product.  I guess one can submit pull requests 
that take the language in the direction one favours though, and 
maybe you do this.

>> questions, but I think your argument would be more effective 
>> if you explained why shipping vibe.d somehow detracts from D's
>
> Because it shifts the focus towards an application area where D 
> will have trouble to gain significant ground. That means the 
> language will be evaluated up to that application area.
>
> There is a limit in the market as new projects will gravitate 
> towards the most promising language in their application area. 
> And there are many languages pitching in the web domain.

It's very hard to know what people ultimately end up doing with a 
tool that you bring into the world, and one may be the master of 
computer science and language design and still be surprised by 
what takes off.  The world is a big place and changing rapidly.  
If one has a set idea about what something should and shouldn't 
do, one may find oneself eventually overcome by Nature, who is 
more powerful - I have given up trying to fight her.

> Which essentially is escapism from a language development point 
> of view. Languages are not judged by their libraries, unless 
> they lack functionality due to flaws in language semantics.

It depends on who is doing the judging, and what they are trying 
to do.  The decision by a commercial user to adopt a language 
framework surely does depend on the cost of accomplishing her 
goals using that framework, and this surely depends for many 
domains on the implementations and libraries available.  It's a 
funny thing I notice people do to pretend that decisions about 
language adoption are based on the merits of the pure language 
itself, when only for a subset (I suspect a minority) is that 
truly the case.

[I am not sure if it is escapism to listen to your market and do 
what you need to to address the biggest concerns].

> This is different in a scripting language which often is used 
> in contexts where you cannot predict your needs ahead of time. 
> I.e. you are prototyping and are exploring new directions or 
> are just covering your needs day by day. If you are doing that 
> in a long running predictable project you are in a bad shape 
> (aka fire fighting).

Fair point, although I suspect this is a feature of the domain 
not the language.  One might write a bond analytics framework in 
C++, but that doesn't mean one knows how it ultimately is going 
to be used.  The world is a big place, and one doesn't 
necessarily understand the needs of others.  Of course it is 
frustrating if the world charges ahead in a direction that one 
doesn't find interesting, but submitting code probably has more 
influence than telling people they shouldn't do this.

This whole scripting vs system language thing suffers from 
reification of a distinction that once mapped to something crisp 
in reality, but no longer does.  AHL (one of the largest 
systematic fund managers) have all their trading systems in 
Python, for example, so the connotations of lack of robustness 
and difficulty of building large and complex systems that perhaps 
were once associated with the idea of a scripting language 
perhaps apply less today.  (Which isn't to say that you will get 
me to love dynamic typing for serious work).

Is Go a scripting language or a systems language?  On the one 
hand, the D front page no longer positions D as a systems 
language (which I think is the right move); on the other, people 
are using it for low-level stuff.  So why get hung up on labels: 
technology is a tool for solving problems, and the question is 
how well adapted a particular tool is to the particular problems 
one faces (and how easily it can get there with a bit of work).

> Adam is a great guy, but he is probably more patient than most 
> with figuring out workarounds ;-).

Yes - which is why he (and his unique kind of way of being in the 
world) is so valuable.  Someone needs to break the ground, and by 
doing so makes it easier for everyone that follows.  See the very 
interesting work on embedded systems for ARM Cortex.  Adam's 
technical contribution is large in itself, but the effect of his 
inspiration on others may well be larger.

>> Similarly the work on ARM/Android/iOS, which seems to be 
>> coming along.
>
> Maybe, I do iOS work and it is very convenient to just use 
> Objective-C++ everywhere I need something that cannot be done 
> in C++. Add to this that Apple keeps mutating their libraries 
> and Apples IDE becomes kind of irreplacable. You need something 
> a lot better than C++ to encourage a switch there...

Let's see what happens - I am very interested to find out.  I am 
looking at a project that might involve D on the server and PC 
client side, and it is very nice to know that by the time I need 
it, probably it will be viable for the analytics on mobile (even 
if you glue it together with something else).


>>> There is a need to move towards something beautiful, and 
>>> that's not in Andrei's vision, but in the original D1 vision 
>>> + the improvements proposed by Bearophile, Timon Gehr and 
>>> others.
>>
>> I appreciate you may not have time, but if you had any links 
>> to stuff if they are gathered in documents rather than myriad 
>> fragments, I would be curious to see.
>
> I don't think so, but it is mostly a fairly standard stance 
> about programming language ideals. (Which C does not adhere to, 
> and D leans heavily on C.)

If you have time, I would very much appreciate any book 
suggestions, and the like.  I am returning to programming after a 
twenty year break, and apparently thinking has moved on a little 
since I was away ;)

> Then rework the memory model, which is a lot of work if done 
> well, to a D3 version of the language.
Have you written anything on what this should look like?

> Fudging it with reference counting hacks makes D not very 
> attractive beyond "compiled scripting", but "compiled 
> scripting" is better off with a good GC than unmanaged memory 
> handling and ref counting by default...

I didn't know anyone wanted to change the defaults, rather than 
to offer some more choices.

> So the proposed solutions have a very low potential for 
> increasing market share.

But do you think that you are looking at it the right way?  Must 
it be the case that we are all in a battle to the death for a 
share of a limited pie?  I personally tend to agree with Peter 
Thiel that it is a destructive and false belief to think that as 
an entrepreneur (whether in the commercial or open source worlds) 
one should think of competition as a positive thing.  It's much 
better for oneself to strive for a monopoly, but a monopoly 
gained through creating something valuable.  And it's probably 
better for everyone else, too.

http://www.wsj.com/articles/peter-thiel-competition-is-for-losers-1410535536

"Americans mythologize competition and credit it with saving us 
from socialist bread lines. Actually, capitalism and competition 
are opposites. Capitalism is premised on the accumulation of 
capital, but under perfect competition, all profits get competed 
away. The lesson for entrepreneurs is clear: If you want to 
create and capture lasting value, don't build an undifferentiated 
commodity business.
...
So a monopoly is good for everyone on the inside, but what about 
everyone on the outside? Do outsize profits come at the expense 
of the rest of society? Actually, yes: Profits come out of 
customers' wallets, and monopolies deserve their bad 
reputation—but only in a world where nothing changes.

In a static world, a monopolist is just a rent collector. If you 
corner the market for something, you can jack up the price; 
others will have no choice but to buy from you. Think of the 
famous board game: Deeds are shuffled around from player to 
player, but the board never changes. There is no way to win by 
inventing a better kind of real-estate development. The relative 
values of the properties are fixed for all time, so all you can 
do is try to buy them up.

But the world we live in is dynamic: We can invent new and better 
things. Creative monopolists give customers more choices by 
adding entirely new categories of abundance to the world. 
Creative monopolies aren't just good for the rest of society; 
they're powerful engines for making it better."

Speaking as a commercial user that has an intellectual curiosity 
about languages, and the enthusiasm of a craftsman for a tool 
that can help do a job well and efficiently, what matters as a 
user of D isn't its market share, but whether one can bet on it 
being around in 5-10 years, whether one can find people capable 
of helping one as one grows, whether its robust enough for ones 
application (not all compiler bugs are equal), power, efficiency, 
productivity, and the existence of and ease of porting frameworks 
needed to accomplish one's ends.

So as regards adoption, the following chart is much more 
interesting to me than market share - and I think it should be to 
you too, perhaps!

https://qph.is.quoracdn.net/main-qimg-ee719ae3a29d4523251255a604a16a6d?convert_to_webp=true

> In fact some of the proposed changes would probably make the 
> language hard to analyze which has a bad effect on future 
> tooling and a programmer's ability to keep a sane model of the 
> language in his head.

Seriously?  One is hardly going to need to bother with the 
allocation stuff and reference counting if one doesn't need it (I 
am not sure what other factors you mean).  And it's the coherence 
or lack of it that leads to difficulty fitting things in one's 
head - if things are based on principles you don't need to learn 
a whole bunch of rules.  I personally found D quicker to pick up 
to reach a level where I can be decently productive than Python, 
and I don't think that would have been true of C++.  So there is 
plenty of spare cognitive budget to spend on a bit of optional 
complexity if it's done right to serve a real need.

Plus I personally wouldn't be short (bet against) quality of 
tooling if you look at how things have developed and what is in 
the pipeline.

> However the "winner takes it all" effect has become a lot 
> stronger now that you have so many excellent free libraries.

I thought languages weren't judged by libraries ;)  But I take 
your point, and agree with it - which is why C++ interop is so 
satisfying to see develop.  And perhaps it is not necessary to 
have libraries written natively, but merely the key components, 
with bindings/wrappers being quite satisfactory for everything 
else.

Empirically though, as someone else here said, the number of 
languages in decently wide usage does not seem to fit your view 
of a winner take all phenomenon (I am not sure if you intend this 
to apply to all languages, or just to systems languages).  That's 
what Knuth called for in the talk I posted from decades back - he 
said that because use cases differ and because of the diversity 
of cognitive styles, one size fits all in language design was not 
the right way.

>> You might as well have said To Honda, Toyota, Nissan, Hyundai 
>> etc that "it's a winner takes all game" when their products
>
> Nah, there is plenty of commercial activity in the car industry.

Perhaps I was unclear.  Japanese auto sector used to be a joke, 
and market share was tiny.  They achieved adoption at the 
fringes, and then used this as a launching pad to move into more 
impressive domains.  Similar stories with newer ways of making 
steel, for example.  Thiel talks about this.  Ie you don't win by 
beating your opponent head-on in the area where he is strongest - 
that's just suicidal.  So in my view it's totally irrelevant to 
speak about what would be needed to get a core heavy C++ guy to 
switch - you want to persuade the guy who has unusual needs, who 
is unhappy with his existing options, who has more freedom to try 
things, and the like.  Then if you achieve big wins in small 
areas, other people will slowly take notice.

> I'm not irritated by it. It just does not represent system 
> level programming, so unless D stops claiming to be a system 
> level programming language (like Go) it should not be the 
> primary long term target. Pervasive reference counting is a 
> scripting language solution.

Okay.  From what I can see D these days claims to be D - not even 
in essence a better C++ (although that is one facet of it).  Not 
quintessentially a scripting language, although it can do that 
very well in many cases, and not quintessentially exclusively a 
systems language.  Does it matter what label you pin on it ?  It 
might be more constructive to say: here is the problem I am 
grappling with in building this embedded system/doing this audio 
processing/etc, and there are these little things in the 
runtime/library etc that get in the way.

> System level programming means you control memory layout, 
> memory usage etc. For instance, in my current project it would 
> perhaps be easier to use ref counting, but since I 
> generate/load arrays that could easily consume 40% of the 
> memory I better be sure that the memory is released before 
> loading the next set. Otherwise real time performance will 
> suffer (audio playback).

Yes - I am just now having the same problem, although luckily for 
me it's not real time, and I can just allocate a static buffer 
once and re-use it.

> That level of control is overkill for processing historical 
> data, but the most promising solution for that application area 
> are distributed cloud solutions. Like Google Big Query that 
> AFAIK can do brute force SQLish quries over very large datasets 
> fast (using some kind of built in query optimization).

There is a lot inbetween the area where Python chokes, and where 
you want to have the complexity, hassle and expense of a managed 
cluster.  Also, in this environment, it's so much easier to 
prototype something where one doesn't require justifying a budget 
than to have to do it in an industrial style at vast scale from 
day one.  I brought up the 8Tb drive as just one straw in the 
wind for something that will surely be playing out in a fractal 
way.

I'll leave it there as per Andrei's request about focusing on the 
hackathon.

Always interesting to exchange perspectives.


Laeeth.