Vision document for H1 2017

[Andrew Browne via Digitalmars-d-announce]

On Friday, 13 January 2017 at 04:02:38 UTC, Jonathan M Davis 
wrote:
> On Thursday, January 12, 2017 21:57:37 Andrew Browne via 
> Digitalmars-d- announce wrote:
>> On Wednesday, 4 January 2017 at 19:22:33 UTC, Andrei 
>> Alexandrescu
>>
>> wrote:
>> > We release a brief Vision document summarizing the main 
>> > goals we plan to pursue in the coming six months. This half 
>> > we are focusing on three things: safety, lifetime 
>> > management, and static introspection.
>> >
>> > https://wiki.dlang.org/Vision/2017H1
>> >
>> >
>> > Andrei
>>
>> Is there a design document for how D will achieve safety with
>> nogc?
>> How does D plan to prevent leaks, use-after-free, double-free
>> bugs when not using the GC?
>
> Part of the reason that we have the GC in D is because of the 
> safety guarantees that you can have with a GC that you can't 
> have with mechanisms like malloc and free. Some amount of @nogc 
> code can be @safe, but some of it will never be able to be 
> @safe. e.g. the very nature of malloc and free makes @safe 
> impossible in the general case.

This is why I ask. Because the vision document says

"2. @nogc: Use of D without a garbage collector, most likely by 
using reference counting and related methods Unique/Weak 
references) for reclamation of resources. This task is made 
challenging by the safety requirement."

It sounds like @safe @nogc code is the goal. I know it is 
challenging, I'd like to know if there is a plan/design for how 
to achieve this.

> It's trivial for a piece of code to free something that's 
> currently in use by other code. If they're constrained within a 
> ref-counting system, then @safety becomes more possible, but 
> even then, when you have to start worrying about stuff like 
> weak references in order to get around circular reference 
> problems, it gets dicey if not impossible to make it fully 
> @safe.

Yep, circular references are a difficulty of ref-counting. Is 
there a plan/design for how to handle this?

> It might be possible to guarantee safety if you have a whole 
> bunch of extra constraints like Rust does with its borrowing 
> stuff, but we're not going to add something like that to D, 
> because it's too complicated on top of everything else that we 
> already have.

So that level of safety is just not a goal of D? If I want memory 
safe code without a GC, should I just use Rust?

> I fully expect that certain idioms will be in place to help 
> maintain @safety in @nogc code, but to some extent, by 
> abandoning the GC, you're abandoning @safety - or at least 
> you're making a lot more of your code need to be @trusted, and 
> you can rely less on the compiler to guarantee @safety for you. 
> Taking the freeing of memory out of the hands of the programmer 
> like happens with the GC is _huge_ in guaranteeing the memory 
> safety of code.
>
>> Will @nogc also have first class support in the language?
>
> And what do you mean my first class support? Some features 
> require the GC, and I wouldn't expect it to ever be otherwise. 
> Giving up the GC means giving up on certain features. We don't 
> want that list to be longer that it needs to be, but some stuff 
> fundamentally needs the GC to do what it does.

That is what I meant. The next questions are examples of what I 
meant.

>> Afaik the GC is currently needed for language features like 
>> array concatenation. Will features like array concatentation 
>> still work with @nogc?
>
> I don't see how it possibly could given how dynamic arrays work 
> in D. It would have to have some sort of reference counting 
> mechanism, which would likely be a nightmare with slicing and 
> certainly does not at all play well with how low level D arrays 
> are. We may very well get some sort of ref-counted array type 
> that has concatenation, but it would be a library construct 
> rather than in the language, because it doesn't need to be in 
> the language, and the built-in arrays would not be affected by 
> it.
>
>> GC allocations have a keyword 'new' (afaik 'new' currently 
>> never means anything other than GC allocation). Will we be 
>> able to do @nogc allocations by the 'new' keyword?
>
> I very much doubt it. Constructing objects into memory is done 
> via emplace, which is a library construct, and there's really 
> no need for it to be in the language. As it is, if we were 
> doing things from scratch, new probably wouldn't even be a 
> keyword. It would likely be a library construct in druntime, 
> because D is powerful enough that new doesn't need to be in the 
> language to do what it does. And in general, at this point, 
> Walter and Andrei don't want to put stuff in the language 
> unless it actually needs to be there. If it can be done with a 
> library, it will be done with a library. The only reason that 
> they decided that we needed some sort of ref-counting mechanism 
> in the language is because they decided that it wasn't actually 
> possible to make it fully @safe without it being part of the 
> language. And even then, I'm not sure that the intention is 
> that the ref-counting mechanism use anything other than the GC. 
> It's not yet clear what it's going to look like, but 
> previously, the talk was using the GC to take care of circular 
> references, which would mean that the memory was still 
> GC-allocated even if it were ref-counted. We'll have to wait 
> and see though.
>
>> Is the same code always expected to work with/without @nogc?
>
> That would depend entirely on the code. 
> std.experimental.allocator has a GC allocator. So, code that is 
> designed around it could work with the GC or without. But the 
> whole mechanism of newing something up and then not worrying 
> about ownership which happens by default with the GC doesn't 
> play at all nicely with how memory has to be managed via other 
> mechanisms like malloc and free. I don't think that it's at all 
> reasonable to expect that code that is written with the idea 
> that its memory will be managed by the GC will also work 
> without the GC. Code that isn't managing memory directly 
> shouldn't care - and that's a lot of code (especially once lazy 
> ranges are in the mix) - but I think that it's pretty clear 
> that there's plenty of code that simply can't just swap out its 
> allocation mechanism and still work properly. In general, code 
> needs to be written with that in mind for it to work, and even 
> then, there are limits given how different various memory 
> management mechanisms are.
>
> - Jonathan M Davis

Yep, I know there are a lot of things that become hard to make 
safe without a GC.
That is why, I'd like to know what is a plan/design for achieving 
@safe @nogc code? What is not planned to be supported?