Are Fibers just broken in D?

[Steven Schveighoffer]

On 4/24/18 3:45 PM, Radu wrote:
> On Tuesday, 24 April 2018 at 16:05:48 UTC, Steven Schveighoffer wrote:
>> On 4/24/18 10:16 AM, Radu wrote:
>>> On Tuesday, 24 April 2018 at 13:36:48 UTC, Steven Schveighoffer wrote:
>>>> On 4/24/18 5:11 AM, bauss wrote:
>>>>> On Tuesday, 24 April 2018 at 07:58:01 UTC, Radu wrote:
>>>>>> On Tuesday, 24 April 2018 at 00:46:39 UTC, Byron Heads wrote:
>>>>>>> [...]
>>>>>>
>>>>>> This is not a fiber issue but a more memory management issue. Your 
>>>>>> run out of address space on win32, the GC will not always collect 
>>>>>> all those 99999 fibers that you allocate in that loop. As an 
>>>>>> exercise replace `auto` with `scope` like `scope foo = new Foo();` 
>>>>>> in that loop - you should see different results.
>>>>
>>>> This shouldn't be a requirement, the 32-bit GC is generally not this 
>>>> bad.
>>>>
>>>
>>> Allocating so many fibers in a loop produces an OOM error on win32, 
>>> that's a fact! Event though it doesn't always happen you often get 
>>> OOM errors with the program above.
>>
>> I'm not saying it doesn't happen, just that it *shouldn't* happen. At 
>> least not for small sized chunks like this.
>>
>> I want to emphasize that the program is allocating and releasing to 
>> the GC 1 fiber at a time -- loop or no loop, this should work (more or 
>> less) reliably, or Win32 has some more serious issues.
>>
> 
> Changing main to
> ---
> void main(string[] args)
> {
>      import core.memory;
>      foreach(ulong i; 0..99_999) {
>          auto foo = new Foo();
>          foo.call();
>          foo.call();
>          if (i % 10000) // <-- this
>              GC.collect();
>      }
> }
> ---
> 
> makes the OOM error go away.

This made it click for me -- VirtualAlloc does NOT allocate from the GC. 
I had mistakenly thought the GC was being used to allocate the stack. 
This means there is little to no pressure on the GC to run a collection 
even though all the memory is being consumed. In other words, the 
runtime has plenty of space to allocate the Fiber class (which is likely 
about 64 or 128 bytes per instance), and is consuming all the memory via 
VirtualAlloc.

I also noticed, Windows default stack size is 32k, not 16k (as it is on 
other systems), so 100,000 stacks in that case is 3.2GB. That's too much 
for sure.

I'll file an issue. We may not be able to solve the problem, but it's 
something we should try and solve.

Thanks

-Steve