What is the right level of abstractions for D?

[Joakim via Digitalmars-d]

On Saturday, 29 October 2016 at 22:38:07 UTC, Danni Coy wrote:
> The bit that sends group 3 people off screaming is static 
> typing, once you have accepted the need to explicitly type 
> everything then Template functions are pretty straight forward 
> in D.
>
>> D can probably do well with groups 1 and 2, but the level
>> of power and expertise that is needed for those lower levels 
>> will scare away
>> people from 3.  Those already using it for 1 and 2 may also be 
>> comfortable
>> with reusing D for scripting, but that's not attracting people 
>> from group 3,
>> ie those who only want something easy to use and don't want to 
>> know the
>> difference between a static and dynamic array.
>
> Again this boils down to static typing, once your group 3 
> programmer has accepted this then is is a minor issue. At worst 
> case I could see a grumble then avoiding static arrays.

As someone who uses all three, I think you underestimate the 
reluctance of many in group 3 to learn enough to drop down into 
the lower-level languages.

> Apart from in application scripting, I have replaced python 
> with D at work. (I was largely using python because my brain 
> and shell scripting are largely incompatible). The reasons for 
> this was mostly because with D I can compile and distribute the 
> binary without having to worry if the right version of python 
> was installed on the target machine.

I have done something similiar with a scripting language before, 
shipping the scripting interpreter bundled with the app, so that 
nothing else has to be installed.

> For this whip up a tool to automate some file processing 
> quickly I
> found D as pleasant as anything else I have used.

I would only use D for such a task too, but I think that is more 
that we are in group 1/2 and are reusing the tool we know.  I'm 
talking about attracting those from group 3.

> The only thing missing are bindings for a high quality 
> crossplatform
> gui toolkit.

Have you tried DlangUI?

https://github.com/buggins/dlangui

I haven't tried it much, so I can't speak to quality, but it 
certainly is very cross-platform.

> Where I see the future potential of D in regards to the levels 
> 1,2 and 3 is having a team where different programmers are 
> working at different levels within the same language on the 
> same project. Take a game engine, typically you have low level 
> engine guys (level 1) some guys on the engine team who aren't 
> quite as good but don't have to be (effectively level 2) and 
> the guys who do the level / ai scripting (level 3). I would 
> quite happily use D as a high level language in a team where I 
> had a good lower level team designing most of the stuff I use 
> with my level of expertise in mind as they designed it.
>
> To this end the really critical things for me are that my level 
> 1 guys can create code that performs in low latency 
> environments without missing deadlines and can present their 
> APIs in such a way that is easy/enjoyable for us level 3 guys 
> to use (see the batch processing thread a little while back).

That's an interesting use, guess Ethan is heading down that road 
with his Binderoo effort.  If D can be used like this for more 
than just games, it would create a nice niche, but I'm skeptical 
of attracting much of group 3.

On Sunday, 30 October 2016 at 23:25:03 UTC, Laeeth Isharc wrote:
> Those categories - I am not sure how well they fit.
>
> When I learnt to program, C was considered a high level 
> language,
>  and now Swift is considered low level.  The world has changed 
> a little, but that isn't my main point.

There were higher-level languages than C even back then, but 
computers were much slower, so C was considered the highest 
practical level.  The difference is that computers are so fast 
now that many can actually use lisp or smalltalk.

> Implicitly in what you wrote is the idea that low level 
> programmers are the ones with real ability,  and people who 
> write in Python might be productive,  but are of a different 
> level of overall ability.
>
> Once that kind of high level / low level mapping to ability 
> might have made sense,  but if it's still useful,  I think it's 
> much less applicable now.   There are plenty of mediocre 
> embedded device programmers,  and plenty of people who used to 
> write in C and now write in Python.   (I am sure ESR still 
> writes in C, but he write about his love for python some time 
> back).

I never mentioned anything like "real ability," whatever that 
means, only that you cannot abstract away the computer as much 
when you need more performance.  Those in group 3 don't need 
performance, so they are abstracted away from types, virtual 
methods, and memory allocation, low-level details that they don't 
want or need to deal with.

Mediocre embedded guys still have to know more about the hardware 
than a Python programmer, that's the knowledge/skill I'm saying 
they have more of.

> And to call python a scripting language is misleading 
> terminology - conventionally so,  but still  true - for 
> example,  AHL,  the large quant money management firm,  wrote 
> their risk management systems entirely in Python. You may be an 
> enthusiast of Fisher Black's Noise paper and think that people 
> in money management are fooling themselves,  and I am 
> sympathetic to some of that,  but my impression is that 
> technically this firm is decent.

Scripting is just how they're named, obviously you can write 
competent software with them too, but I'd be surprised if AHL's 
software were that fast.  Maybe they don't need it to be 
speedier, so python is best for them, but I never said anybody in 
group 3 is not technically "decent."

> And everything gets more mixed up when you can compile a Ruby 
> dialect and have it appear at the top of the performance tables.

Which dialect is that?  I did mention that group 2 is going 
AoT-compiled and speeding up, so it is possible with more work to 
speed up parts of the higher-level languages, but those are 
subsets.

>  It was a scripting language before,  and now it's not?  (it's 
> control and level of abstraction rather than performance that 
> distinguishes level of a language,  but in the past these 
> things went together).

You probably can't use rails with your ruby dialect, with its 
extensive use of eval(), so there go most of your users.  
Abstractions often cost performance, which is why those 
higher-level languages are slower.  D tries to keep its 
abstractions as low-cost as possible, similar to C++, but that 
comes with complexity that group 3 doesn't want to deal with.

> It seems to me you are reifying your class structure of 
> languages when I am not sure it is a good representation of how 
> things are.

I think that hierarchy is pretty uncontroversial nowadays.

> The reason scripting applications don't use D if it's not 
> already used for other things is libraries and polish.   D has 
> great python interoperability and also a nice package manager.
>  Well,  try compiling a program as a python library depending 
> on vibed using dub.   It won't go very well because the fPIC 
> isn't propagated and you might need to compile in a little C 
> code got the constructors.   And what needs to be done isn't 
> completely documented anywhere. So at this point,  it's a less 
> appealing proposition because for the people that currently use 
> D,  these things don't bother them as much and because it's 
> still a small community with a lot of work to do. John Colvin 
> is working on it,
>  and maybe it will be fixed soon - because it did bother me.
>
> But this isn't a consequence of the Platonic essence of D as a 
> language.   It's merely contingent on the particular stage of 
> development and it couldn't have been otherwise because you 
> have to get the foundation right before putting sugar on top.

Libraries and polish are part of the problem, but not what I'm 
talking about.  You cannot make every power tool as easy to use 
as a screwdriver.  You want performance, you have to pay for it, 
by learning how to get it.  D makes an effort to pay less and 
does a remarkable job, but I think group 3 is out of reach.

> The experience of social learning is that every person that 
> follows a course somewhat mysteriously makes it easier for 
> those that follow.   It's not only mysterious,  because the 
> signposts and guides get better too.   D is not an easy 
> language to learn,  but even today it's far from exceptionally 
> difficult,  and it will get easier with time.
>
> If you want a C and C++ ABI,  want to have control over memory 
> and to go down to a low level if you need it,  but are someone 
> in a position where you need to get stuff done,  and don't 
> think modern C++ is the answer,  what choices do you have?  Not 
> all that many.
>
> And I have thought for a while that people were recklessly 
> squandering performance and memory usage.   That's not good 
> craftsmanship - it might be necessary to make compromises in a 
> practical situation, but it's hardly something to be proud of,  
> as people almost seem to be.   What Knuth said in his paper and 
> speech is not what people take his words out of context to 
> mean,  and on the other hand look at the everyday experience of 
> using applications written by people who follow this philosophy 
> to see that there must be something wrong with it.

As I've said in this forum before, if you want performance, I 
wouldn't recommend anything other than D.

> A language really takes off not because of something it started 
> doing that it didn't before,  and all of a sudden everything is 
> a success.   It takes off when you have the seeds of something,
>  and external conditions change to the point that suddenly it 
> starts to become a much more favourable environment for 
> language adoption.

I think the recent move towards performance, because of the more 
constrained mobile environ, is that change.

> Herb Sutter wrote a long time back about the end of the free 
> lunch from Moore's Law.   And its already visible today - 
> somehow one ends up being CPU and memory bound more than Guido 
> would have us believe (I am not sure quite often that Python is 
> fast enough).   It's why I adopted D, and another chap from 
> hedge fund land who has spoken at dconf.   He said to me this 
> year that things had a bit the feel of python scene in the 90s 
> before it really took off.
>
> Dataset sizes keep growing.  And I don't know,  but it seems to 
> be that the ACM paper I posted earlier about the consequences 
> of new fast non-volatile storage technology is right - storage 
> won't be the bottleneck for much longer, and increasingly CPU 
> will.   Intel described the technology as the most significant 
> development since the Internet.   I am not sure that is merely 
> hyperbole.
>
> When you see a change in relative price of the sort they are 
> talking,  and it's something so basic,  then everything changes.
>  Not overnight,  but over the course of the next five to ten 
> years.

Yes, some big changes are happening, that favor group 1 again.  
It's why we see a renewed focus there, with new entrants like 
Rust, Swift, and Go, and C++ finally reforming.

> I can't imagine this shift will be bad for D,  and if that's 
> the case then the pattern of people drawn to explore D at the 
> margin won't necessarily fit your categories.   I should think 
> the determinants will be more things like how much work and 
> data do you have in relation to easily available processing 
> power (yes,  the cloud is cheap,  but there are institutional 
> and contractual barriers to moving data outside the firm,  and 
> these change slowly), how open are you to new technologies,  do 
> you have the authority to try it yourself without persuading a 
> committee,  can you and your people learn something new without 
> extensive training courses,  are you addicted to depending on 
> support,  can you already program in C?  And so on.

They still fit the categories, it's just the relative importance 
of each is changing, with the broader patterns you point out 
favoring group 1 more.

> So look at recent adopters - that probably gives you a little 
> idea about what's coming.
>
> It seems to me the recent low level improvements might be seen 
> as just a general process of maturation that is reflected now 
> in lower level stuff,  previously in documentation 
> improvements,  and tomorrow in something else.   It's not like 
> there is a central direction of any effectiveness on what 
> people work on.

There is a central direction: what the co-architects work on, 
which has recently been @nogc, C++, and safety.  All are geared 
towards group 1, as safety has become a recent concern for group 
1.

> A push for C++ interoperability is a good thing,  but 
> strategically what has that to do with being low level?   It 
> makes it easier to use C++ libraries,  which is also good for 
> scripting...

It is far more likely for a group 1 user to bind to a C++ library 
than a group 3 user, though it potentially helps him too.

I'm not complaining, D's spot between group 1 and 2 is great for 
me.  I have also argued against those who think D should 
specialize even more, by narrowing in on some niche within those 
groups, like games or webdev.  I think some thought has to be put 
into which of those three larger groups D is going after, and 
tweak some of the messaging because group 3 is out of reach.