Reimplementing software building blocks like malloc and free in D

[Mike Franklin]

On Saturday, 11 August 2018 at 18:59:00 UTC, Aruna Maurya wrote:

> 1. If it would have been C++, the above can be implemented by 
> using the mmap() syscall to                  allocate memory 
> from the heap. However, something that I am not able to 
> understand in the idea description is 'runtime hooks'.

The compiler recognizes certain expressions and lowers them to 
functions in the D runtime that we call runtime hooks.  See 
https://wiki.dlang.org/Runtime_Hooks for an unmaintained, but 
still relevant, list of some of them.

For example, the expression `cast(int[])a`, where `a` is an array 
of another type will get lowered to `_d_arraycast`.

We're trying to rewrite many of these hooks as templates.  D has 
changed quite a bit over the past few years, and at the time the 
existing runtime hooks were written D didn't have templates and 
many other features.  See 
https://github.com/dlang/druntime/pull/2264 for an example 
converting `_d_arraycast` to a template.

Many of the existing runtime hooks leverage software building 
blocks like `memcpy`, `memcmp`, `malloc`, `free`, etc. in their 
implementation.  If the runtime hooks are implemented as 
templates, we will have access to the type information and other 
information at compile-time rather than run-time.  This will give 
us opportunities to specialize implementations at compile-time.  
See https://github.com/JinShil/memcpyD/blob/master/memcpyd.d for 
a proof of concept exploring such opportunities for `memcpy`.

> 2. Also as far as I can understand, this idea is about 
> implementing the above from scratch, and not tweaking the 
> existing codebase. Do let me know if I am wrong.

I'm the one who proposed the idea, and, yes, I envisioned 
implementing these building blocks from scratch, translating an 
existing implementation from another language (watch the 
license!), or implementing an existing published algorithm in D.

There might even be a way to leverage implementation code in 
https://dlang.org/phobos/std_experimental_allocator.html, but if 
that code were used in the D runtime, it couldn't have 
dependencies on Phobos (D's standard library), so it'd probably 
have to be modified to make it free-standing.  I don't know, 
though, I haven't looked into it in any detail.

I didn't envision the participant modifying any existing code.  
In the case of `malloc`, `realloc`, and `free`, I'd just like to 
have an idiomatic D implementation that we might be able to 
leverage when re-implementing the runtime hooks as templates so 
we no longer have to depend on the C standard library.  I think 
implementing them in D will make D more portable to other 
platforms, including freestanding bare-metal platforms like 
operating systems or microcontrollers, and D's unique feature set 
may provide opportunities to improve upon existing 
implementations in terms of performance and compile-time 
guarantees.

It's difficult for me to articulate all this, as it's a complex 
vision in my head, so if the above just raised more questions, 
keep asking.

Also see https://wiki.dlang.org/Memory_Management for some 
perspective, especially if you're new to D.

The ideas on the wiki page are just to plant the seeds of 
creativity.  Feel free to deviate from the idea as you wish, or 
submit a proposal formulated from your own ideas.

Mike