Reimplementing software building blocks like malloc and free in D

[Mike Franklin]

On Sunday, 12 August 2018 at 04:16:24 UTC, Aruna Maurya wrote:

> So there are essentially two requirements here. Building the 
> memory management functionalities from scratch in idiomatic D 
> and implementing the existing runtime hooks as templates. The 
> latter isn't quite a part of the idea as mentioned on the page.

I'd say there is only 1 requirement:  implementing `malloc`, 
`realloc`, and `free` in idiomatic D without any dependencies on 
other libraries from other languages.

The "idiomatic D" part means you don't even have to name them 
"malloc", "realloc", and "free", nor use the same type-unsafe 
signatures found in the C standard library.  Do what makes sense 
for D, for example, you may want to implement an overload set 
like `T[] heapAllocate(T)(size_t length)` and `T 
heapAllocate(T)()`, or something totally different.  You may even 
want to provide some other features for gathering runtime 
statistics on the heap.  The world's your oyster here.

The only reason I mentioned the runtime hooks is to help make the 
case for why these D implementations are needed and provide some 
perspective about how they may be used.  I don't envision the 
Autumn of Code participant learning about the compiler/runtime 
interface and refactoring the D runtime with the D 
implementations (unless they really want to).

Regardless of whether we decide to incorporate these D 
implementations into the official D runtime or not, they will 
still be worth gold to those who desire using D in freestanding 
systems programming, so don't concern yourself too much with the 
d runtime; just make a kick-ass dynamic heap allocator in D, and 
people will use it.


> Also I see in one of the above replies that malloc and free 
> have been already implemented. So is that off the shelf or 
> still available for implementation?

I wasn't aware of Eugene's implementation.  At first glance it 
looks real nice.  It appears to have some dependencies on other 
features of his Tanya library, so at a minimum, those will have 
to be removed or replaced.

But, how does it perform?  In order to make the case for using 
these D implementations in druntime, we'll have to gather some 
metrics on the implementations in terms of performance, memory 
fragmentation, etc. and ensure they are on par with the C 
standard library implementations or alternative implementations 
like jemalloc and tcmalloc?

Also, can Eugene's implementation be used in `@safe` code and is 
it compatible with the DIP1000? (See 
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md and the 
-dip1000 compiler switch at 
https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md)

Perhaps the D implementations can be enhanced to provide 
compile-time or runtime tuning parameters to achieve better 
performance in different use cases.

Perhaps, with Eugene's permission, you can use his implementation 
as a base, take some measurements, and see if it can be improved 
upon with either more features or better performance.

I'm sorry if I'm just muddying the waters with all this talk.  
All I want is a high-quality, idiomatic D implementation of a 
heap allocator that I can use in my own freestanding software and 
as a substitute for the C standard library implementations in the 
D runtime.

Mike