App hangs, GC.collect() fixet it. Why?

[Bastiaan Veelo]

On Monday, 28 September 2020 at 21:58:31 UTC, Steven 
Schveighoffer wrote:
> On 9/28/20 3:28 PM, Bastiaan Veelo wrote:
>> I’m leaning towards ditching the memory mapped I/O on the D 
>> end, and replace it by regular serialisation/deserialisation. 
>> That will be a manual rewrite though, which is a bit of bummer 
>> as memory mapped files are widely used in our Pascal code. But 
>> this will probably give the best end result.
>
> 2 things:
>
> 1. I agree this is the answer. If you ever ditch the old Pascal 
> code, then you can reactivate the memory-mapped code.
> 2. You can possibly do the translation outside of your 
> programs. That is, it wouldn't be entirely impossible to simply 
> have a process running that ensures the "D view" and the 
> "Pascal view" of the same file is kept in sync. Then you can 
> keep the memory mapped code the same, and just define sane 
> structures in your D code.
>
> If you aren't required to have both Pascal and D programs 
> reading and writing the file at the same time, this shouldn't 
> be a problem.

There is no need to run both versions concurrently. The issue is 
that design offices typically maintain a library of past designs 
for as long as they are in existence, to build new designs off 
of. So being able to read or import the files that were written 
with an ancient version of our software is very valuable. Our old 
compiler offered two alternatives for file i/o: one where all 
elements are of the same type, the other one (memory mapped 
files) being the "only" option for files of mixed type. Ideally, 
the structs that are used for i/o do not have any pointers in 
them, and certainly in the more recent file versions that would 
be the case. In older versions that might not be the case; then 
the pointers obviously would be given meaningful values after the 
structs would have been read back in. These cases we would be 
able to work around, though, by converting the old structs to new 
ones upon import.

> BTW, one further thing I don't understand -- if this is memory 
> mapped data, how come it has issues with the GC? And what do 
> the "pointers" mean in the memory mapped data? I'm sure there's 
> good answers, and your actual code is more complex than the 
> simple example, but I'm just curious.

The main problem is that the transpiler doesn't know which 
structs are used for i/o and would need 1-byte alignment, and 
which structs have pointers into GC memory and must not be 1-byte 
aligned. The alternative to switching to 
serialisation/deserialisation is to stay with the automated 
translation of the memory mapped file implementation, not 
automatically 1-byte align every struct but manually align the 
ones that are used in i/o. This is however sensitive to mistakes, 
and the translated mmfile implementation has a bit of a smell to 
it. It is also not portable, as it uses the WinAPI directly. 
Still, it may be the quickest route to get us back on track.

I am very glad to have identified the problem, and there being 
ways to deal with it. I just hope this will be the last big 
hurdle :-)

-Bastiaan.