A betterC modular standard library?

Sun Dec 18 10:02:58 PST 2016

On Sunday, 18 December 2016 at 15:18:37 UTC, Andrei Alexandrescu 
wrote:
> On 12/18/16 4:26 AM, Ilya Yaroshenko wrote:
>> Hi,
>>
>> Who is interested in betterC _modular_* standard library?
>> I am planing to make libmir org a community for it.
>> Thought and concerns?
>>
>> We already have better `cpuid` and better `random` packages. 
>> The betterC
>> std.range and std.algorithm analogs would be released with new 
>> ndslice
>> implementation. Mir's algorithm would be faster then Phobos 
>> and will
>> generate less template bloat. Then lightweight threads, 
>> multithread
>> GLAS, matrix inversion. Fastest I/O and http2 ...
>>
>> (they all are betterC modular libraries)
>>
>> Andrei and probably 90% of existing D users don't want Phobos 
>> to move
>> this direction.
>
> Please do not misconstrue what I said.
>
> You have my support to improve the state of affairs with the D 
> core runtime and standard library, and have had it from day one.
>
> With regard to new functionality that supplants existing pieces 
> such as cpuid_v2 or random_v2, definitely. We can and should 
> add that and many other great things to phobos. Go for it.
>
> With regard to avoiding to link druntime, we should build the 
> argument better. Is all of druntime bad, or just parts of it?

DRuntime is good for applications. But writing multi-language 
libraries with DRuntime is bad idea. The last paragraph in [1] 
describes why DRuntime can not be used for GLAS.

I never said we should "remove GC" or "remove runtime". We just 
need to provide a clear way for betterC subset and modularise 
library. DRuntime and Phobos are well designed hight level 
libraries.

> E.g. if something does not link the GC but does use vtables, is 
> it good/bad, and to what extent?

Very good point. First, I don't see a real reason why we need 
DRuntime for vtables (classes and interfaces). LDC allows to use 
vtables without DRuntime (only with extern(C++), hehe). D classes 
can be constructed on stack or with allocators. The only problem 
is vtable initlization, which requires `typeid` (TypeInfo 
object). I am not a OOP expert, but it looks like we can have 
extern(C++) classes without DRuntime if we solve problem with 
initilization.

> These issues are not entirely clear to me and it would be great 
> if you explained them better. I definitely need to be better 
> educated on this. What I do know is making "don't use druntime" 
> a goal in and of itself is not the most useful way to frame the 
> problem.

0. Linking compatibility. Assume A has builded a betterC library 
AA using GDC version XXX and B has builded a betterC library BB 
using LDC version YYY.

Question: how mister C can use BB and AA together in his 
C-library if both this libraries depends on different DRuntimes 
(compiler and version)?

1. Different mangling names for DMD and LDC. mir-cpuid provides 
extern(C) interface for all functionality except few very low 
level struct declarations. The general problem is that we do not 
have binary compatibility where we may have it. And it is not 
clear where we already have it.

2. Any function may throw an Error.

3. Any throwable is not a part of betterC. Phobos core team 
rejected approach to replace RangeErrors with asserts or another 
function declaration. Possible alternative is __range_assert() or 
a library function.

4. Threads are GC oriented:
     1. They are classes (they can be structs like allocators)
     2. They are integrated with GC. This requires additional 
computation time if I am not wrong.
     3. They are not replaceable. Phobos multithread modules can 
not use something else.

5. Mutexes are GC oriented:
    ditto 1. and 3.

6. Intrinsics are compiler magic without specification. It is not 
clear what it is intrinsic, are they inlined, where their bodies 
are defined, how/if they works in CTFE. LLVM provides a list of 
intrinsics [3], this is very useful to write a betterC libraries.

... and many other ... They all are not clear for me too.

-------------
A solution for mister C:
Mister A and mister B should use a runtime which has either a 
small stable extern(C) interface or all its functions are 
inlined. All other functions can be a dub packages.
-------------

Proof:
cpuid: extern(C) interface with aliases.
a fork of aligned  mallocator for GLAS: all functions are always 
inlined.

> With regard to changes that break the entire fabric of Phobos 
> (such as: we must eliminate random-access ranges), this is 
> definitely more like simple survival rather than things I want.

I don't want eliminate R-A-R now. But we can change our Range 
concept and provide alternative module as dub package, which will 
be announced as a future replacement (say, after 5 years). But 
duplicated sorting implementation with cursors (lightweight 
iterators) in the same module is weird. It is better to continue 
this discussion after the cursor based implementation will be 
ready for your review.

> With regard to properly attributing credit for good work, I 
> believe this is important and I commit to do anything 
> reasonable for making it happen.
>
> I kindly advise you to better understand what you are trying to 
> do, and how to do it to maximize impact. It seems you are not 
> entirely clear on what you want to do and what the best route 
> is, as shown by you retiring your std.experimental.ndslice work 
> after you've had full freedom on what should go in it.
>
>> In other hand I need a commercial attractive D infrastructure 
>> for large
>> and heavy system projects. There is no commercial perspective 
>> for me to
>> contribute to Phobos because:
>>
>> 1. Phobos version depends on compiler version. Delay with LDC 
>> release is
>> too large. It should / can be one day.
>
> Then a good way to help this is improve the LDC lag.

I mean lag between new Phobos PR and LDC release. This includes 
time for PR review, DMD release, LDC release. Libraries should be 
separated from a compiler, IMHO.

>> 2. We can not do hot fixes in Phobos without compiling and 
>> deploying
>> patched libphobos. Mir's DUB package are much more flexible.
>
> The reverse of that is less stability.

A hot fix can be added in local repository without creating a PR. 
DUB allows to do it.

> The good part about merging your work in Phobos is you maximize 
> impact and visibility.

Yes, but merging existing libraries to Phobos will slowdown other 
projects as DCV. Also  I will spend a lot efforts on stuff like 
repositories synchronisation. Plus it makes my users unhappy: 
they will need to switch from Phobos to Mir if they want to use 
recent addition.

> These seem to me artificial arguments. Yes, being preoccupied 
> over quick deployment may make sense in the initial stages when 
> there's no stability and the design may take any direction. 
> That should get less necessary once some abstractions are in 
> place.

C++ adds new APIs one time per few years. D add few changes every 
month.

Phobos and DRuntime does not behave as traditional standard 
system language libraries. They are more like a scripting 
language heaps which are grows very fast (like scipy).

I wan't a D Standard Library to be exist too, but it should be 
slim, has better backward compatibility, ideal backward binary 
compatibility, has good low level specification and 
documentation. Large hight level library like Phobos can not do 
it with community support only.

> Organizations that are interested in patching minute fixes may 
> build their own standard library (and sometimes compiler). This 
> is routinely the case with C++ at Facebook, Google, and in all 
> likelihood other large-scale tech shops. It would be tenuous to 
> argue that this should be the default way to go about things 
> for everyone. I don't understand why you consider it an 
> essential matter.

Good point. Why not to be more user friendly for large-scale tech 
companies then?

If we can show that merging too LDC requires less efforts then 
upgrading from g++ to clang++ ...

>> 3. Dependencies should be clear. Modularity is a proper way 
>> for large
>> std library. In phobos everything integrated with everything. 
>> DRuntime
>> -> Phobos abstraction is weird for betterC because system 
>> modules can
>> depends universal algorithms, but universal algorithm are more 
>> portable
>> if they have not system dependencies.
>
> I read this three times and cannot understand it. At any rate: 
> DIP 1005, which is partially motivated by our discussions, 
> should allow us to make dependencies as simple and clean as 
> function-level.

The  DIP 1005 idea is good. But this will fix few percentage of 
existing dependency ambiguity. Plus it does not solve the 
dependency problem, it just makes it slightly more visible.

>> 4. Phobos does not provide and will not provide betterC 
>> guaranties. If
>> something works for betterC mode now it may not work in the 
>> future release.
>
> Yes, we can definitely improve this.
>
> We (the Foundation) are all for making things better for 
> achieving the things you want to achieve. At this point D needs 
> unity and rallying under a shared vision, not balkanization in 
> several special interest groups. There are so many great things 
> you can do starting literally today to make things better for 
> everybody using D, and why in the world would anyone want to 
> stop you. I strongly believe your participation to the Standard 
> D language and library will be vastly better for you personally 
> and professionally.
>
>
> Andrei

I will continue in the answer to Walter.

Thank you for the answer (it is hard to understand me because 
English and other reasons),

Ilya

[1] 
http://forum.dlang.org/post/jwmzvfashovivqzamxrq@forum.dlang.org
[2] 
http://forum.dlang.org/post/jneokbosxeatvkppbuvf@forum.dlang.org
[3] http://llvm.org/docs/LangRef.html#intrinsic-functions