How to debug long-lived D program memory usage?

[Alex]

On Friday, 19 April 2019 at 03:27:04 UTC, Adam D. Ruppe wrote:
> On Friday, 19 April 2019 at 02:58:34 UTC, Alex wrote:
>> Curious, what are these programs?
>
> A terminal emulator gui (like xterm), a detachable terminal 
> emulator (like gnu screen), a slack client, an irc client, and 
> a bunch of http servers including d doc search, a work program, 
> and a personal utility.
>

Ok, nothing useful ;) I was thinking you might be doing stuff 
like running a security system that did computer vision, or some 
type of advanced house monitoring and control(voice activated 
doors or something) ;)

Could you not, as a quick fix, just reboot and automate 
restarting those?

Maybe design an auto-restart which saves the state, shuts itself 
down and then relaunches itself and loads the data?

This could be done as a fail safe when memory consumption get's 
too high.

> All of them would show growing memory time, some worse than 
> others. You can see a lot of difference in them - gui programs, 
> terminal programs, network server programs. But, I did write it 
> all myself, so it could be a mistake I keep on making.
>
> So far, I basically tracked down the terminal emulator things 
> to being inefficient scrollback storage. I made the structs 
> smaller and limited the amount saved more than before and cut 
> this by half. The ddoc search was keeping the index in memory, 
> that's fixed, but it still shows growing usage over time. Of 
> course, restarting that is trivial if need be, but still, I 
> wanna make sure I am doing it right too - especially if it is 
> one of my underlying libraries to blame.

Gonna have to be one of those things you track down because it 
could be something as simple as the GC or something more serious.


>> You might have hook in to the GC and just write out stats, I 
>> believe there is a stats collector somewhere though.
>
> Yes, indeed. I am starting to make serious progress now - 
> mostly the fault is me storing excessive histories 
> inefficiently. Should have been obvious in retrospect, but I 
> didn't realize just how much overhead there was in my designs!


D should have a very good memory statistics library built(I guess 
it has something with the switches)... since it should have no 
issues tracking memory usage.

Every memory allocation must have a corresponding deallocation 
for stable programs. All allocations and deallocations have 
specific file locations or occur in the GC.

I don't see why it would be difficult to monitor this stuff. As I 
mentioned, I generally never use new precisely so I can track 
this stuff myself and have a nice printout of memory usage when I 
need it and even verify the net memory allocation 0 zero on 
program exit. I haven't messed with the GC but I imagine it too 
shouldn't be hard to add the info too.