[GSoC] Trying to find a good topic

[Mike Franklin]

On Sunday, 24 March 2019 at 23:26:05 UTC, Dan Printzell wrote:

>> # Suggestion 1 - Replace Runtime Hooks with Templates
> This is something I would like to do. As this would help with 
> both
> writing a new bare metal runtime and it could help the current
> druntime. I can see that this idea would probably be the 
> easiest to
> plan for as you could plan for it to be like X amount of runtime
> symbols per GSoC milestone, or something simular to this.
>
> This could be better than writing a new bare metal runtime as 
> it would
> decrease the amount of code needed to get something to at least
> compile.

Indeed!

There are some dragons to slay, though...

1) This will require familiarizing oneself with both DMD and 
druntime, as both will need to be modified simultaneously.  If 
you intend to make pull requests to these repositories, please be 
aware that there currently isn't a great way to test two 
interdependent pull requests that reside in both DMD and 
druntime.  I recommend the following procedure, but there may be 
a better way:
   1. Add the template to druntime with unittests to verify the 
implementation.  This PR should be merged first.
   2. Modify DMD to lower expressions to the new template.  
Assuming there is already sufficient test cases added to the test 
suite, this should require additional tests; the real test is to 
ensure nothing breaks. This PR assumes 1. is already merged.
   3. Submit a PR to druntime to remove the old runtime hook.

2) I described this in more detail in the last paragraph here 
(https://forum.dlang.org/post/hequvmostuhwnvcgrcge@forum.dlang.org), that the compiler is currently lying to us about `@safe`, `pure`, and `nothrow` guarantees.  When the runtime hook is converted to a template, the template will go through the semantic phase in the compiler, and the author will be forced to suddenly be honest.  It could be quite difficult to make this work without breaking code or without hackish workarounds to avoid said breakage.

As you alluded to, this can be done incrementally.  It may be too 
large a project to tackle all of druntime, so maybe a good GSoC 
project would be, for example, to just get all array features 
finished.

I recommend not worrying about getting things to work with 
-betterC at first; it just adds additional complexity and 
controversy.  If you can first succeed without -betterC, you can 
then make additional changes to make the templates work in 
-betterC.  If you submit PRs to druntime and DMD, this will also 
make the PR review go much smoother.

> I still have to think about if there is a smart way to allow
> "druntime-lite" to be used as a foundation for druntime. 
> Because if
> the new runtime would only be a separate runtime it could 
> become a
> annoyance if you need to update two runtimes for each runtime 
> change.

Annoying indeed!  I don't think it's necessary to have two 
different runtimes with duplicate implementations in each if 
druntime is modularized and the dividing lines between modules 
are well thought out.  Having worked with DMD and druntime a 
little and having given this some thought over the years and 
recently, I'm feeling more confident in something like the 
following:

1) - No-dependency D library (Basically, Suggestion 2 below) -  I 
have more to say about this below.

2) - Level 0 druntime module(s) - Imports (1), but additionally 
adds compiler intrinsics and runtime features that don't require 
dynamic memory heap memory allocation.  I envision these runtime 
features to be common to both bare-metal and OS platforms.

3) - Level 1 druntime module(s) - Imports (1) and (2), but 
additionally adds runtime features that require dynamic heap 
memory allocation and potentially runtime initialization and 
deinitialization, which seem to currently require dynamic heap 
memory allocation.  This would still be quite suitable for some, 
if not most, bare-metal platforms.

4) - Level 2 druntime module(s) - Imports (1), (2), and (3), but 
finishes off the runtime with classes, GC, and threads.  This 
could still be used for some bare-metal platforms, but would 
basically require implementing a mini OS inside the runtime 
itself.  For OS platforms, one just needs to tie into the OS 
implementations as it is currently done.

There are probably hundreds of different ways to organize this.  
For example, the compiler intrinsics and lowerings could be 
separate modules/libraries, threads and fibers could also be 
their own modules/libraries, etc..  The basic idea, though, is 
that users, and the compiler itself, can import just the language 
features they need or want, in a pay-as-you-go fashion.

Also, with design by introspection, even imported modules can 
make decisions at compile-time about what is or isn't needed as 
is currently done with `ModuleInfo`, `TypeInfo`, and `Throwable`. 
  It's important to do this tastefully, however, as it could 
easily turn into a monstrosity if taken too far.

This still needs a lot more thought and work, but that's where 
I'm currently at.  I'm just brainstorming, anyway. I'm not sure 
how something like this would integrate into the existing 
ecosystem.  A good strategy that leverages incremental change 
will probably find more success.  I think Suggestion 2 is a good 
start.  It would help set a precedent and test the palatability 
for more modularity in the future.

>> # Suggestion 2 - Create a No-dependency D Library
>
> One question that need to be decided for this type of library 
> would be
> where should it live inside the D ecosystem, and would this 
> mean that
> the algorithm would live in two places, one copy inside phobos 
> and one
> copy inside this library.

Yes, there are definitely architectural challenges here.  And 
then, worst of all, you'd have to figure out what to name it.  
Deimos is taken, so if my astronomy is correct, we're out of 
moons.

I suggest it be its own library independent from Phobos, 
druntime, and DMD; I think that would be the most scalable.  
Phobos, druntime, and DMD could all privately import and forward 
to it, and users wouldn't even know anything happened.  It could 
also be published as a separate library on DUB for -betterC and 
bare-metal programmers to utilize without having to import Phobos 
and druntime.

DUB could actually be utilized to great effect for all of D 
development if it could just be made into something that the 
entire community could get behind.  In the end, DMD is just 
another D program, and Phobos and druntime are just D libraries.

> It would be nice to try and design a module runtime, but the 
> question
> is how this could be limited into something that I could do in 
> about 3
> months, with clearly defined milestones.

Understood.  There's an awful lot to sort out here. I suggest 
just focussing on few modules from Phobos and/or druntime (e.g. 
std.meta, std.traits, core.attribute, etc...).  Create a DUB 
library and GitHub repository for the GSoC project.  After GSoC 
keep working on it.  It will still be quite useful to -betterC 
and bare-metal programmers.

In the longer-term, it could eventually be integrated into the 
larger D ecosystem and hopefully transferred to the D Language 
Foundation for inclusion in Phobos, druntime, and DMD and 
continued maintenance.  I'm still not sure what that looks like, 
but regardless I think it is a worthwhile project in and of 
itself.

>> # Suggestion 3 - Port the Entire Runtime to [...]
>
> Yes I agree, this would require both suggestions or at least
> suggestion 2. The reason why I have not ported druntime to 
> PowerNex
> yet is because I do not want to sit and add tons of stub & 
> wrapper
> functions to get it to link.
>
> This would be a future goal and not a GSoC goal.

Yeah, Suggestion 1 should significantly reduce, if not eliminate, 
the need for all of those hacking workarounds that are currently 
required just to get a build.  Take a look at the changelog for 
2.079, though, to see what improvements you might be able to make 
today to your bare-metal runtime - 
https://dlang.org/changelog/2.079.0.html#minimal_runtime

>> # Suggestion 4 - Replace Software Building Blocks with
>
> I guess I'm not like most (after all I'm writing my own OS from
> scratch :P), because I see the value of having D implementation 
> of all
> these functions. Basically all bare metal code require these 
> function
> in one way or another. I would say that these are a requirement 
> for a
> bare metal runtime.

Good! We can continue to utilize libc's implementations, but then 
we don't get all of the advantages (design-by introspection, 
CTFE, memory safety, etc...) that could potentially improve 
performance and reliability of the software as a whole.  After 
the runtime hooks are implemented as templates, I would hope that 
we could one day use something like `memcpy(T)(scope T* dest, 
scope const T* source) @safe pure nothrow` instead of 
`mempcy(void* dest, const void* source, size_t n)` in druntime's 
implementations.

If it's not something you're interested in, I hope there are 
others that can see the value in such a thing and will find it to 
be a rewarding project for themselves.  I think it would be a 
great project for a newcomer to D as it doesn't require mastery 
of the language, and the semantics of those building-blocks are 
well-understood.  We won't see the benefit of using these D 
implementations, though, until the runtime hooks are converted to 
templates.


Looking forward to hearing more about whatever it is you decide.

Mike