Either I'm confused or the gc is

H. S. Teoh hsteoh at quickfur.ath.cx
Sat Oct 24 15:46:11 UTC 2020


On Sat, Oct 24, 2020 at 02:21:52PM +0000, donallen via Digitalmars-d wrote:
[...]
> The detective in me couldn't resist (I really need a Peugeot and a
> basset hound) and so I did some more work on this despite not really
> having the time and I have made some progress.
[...]
> Something has caused sqlite to end the return of rows prematurely. But
> next_row_p did not panic, which it will if the the return code from
> sqlite3_step, which it calls, is other than SQLITE_ROW or SQLITE_DONE.
> Since next_row_p just returned 'false', sqlite3_step must have
> returned SQLITE_DONE. sqlite returned 231 of 344 rows but is acting
> like everything is normal.
[...]

Very interesting finding indeed!


> Then I had another of my hunches. I re-wrote the line containing the
> call to format like so:
[...]
>                 children[i].path = (account.path ~ ":" ~ children[i].name);
[...]
> This does not panic! In theory, this should consume the same amount of
> GC memory as the format call (though I have not looked at the format
> code; there may well be intermediate allocations as it does its work).

Am I right in assuming that account.path and children[i].name are both
of type `string`?  As opposed to some other object that defined a
toString method?

AFAIK, format does allocate intermediate buffers for its work; that may
explain the difference you observed here.


> Now I decided to instrument the code so I could determine whether GCs
> were occurring as a result of the call to format:
[...]
> Running this produces:
> dca at pangloss:~/Software/newcash_d/verifier$ ./verifier
> "--DRT-gcopt=profile:1" /tmp/Finances.newcash
> walk_account_tree: bad child index, 231, 344, :Assets:Investments:Equities
> and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade,
> 33, 34
> dca at pangloss:~/Software/newcash_d/verifier$
> 
> This proves that at least once during the loop that failed, a GC
> occurred as a result of the format call.
[...]

Here, there appears to have been 33 GC collections before the format
call, and 34 after. Am I reading this right?


> Next I tried uncommenting the concatenation line and commenting out
> the format call. That produces:
> dca at pangloss:~/Software/newcash_d/verifier$ ./verifier
> "--DRT-gcopt=profile:1" /tmp/Finances.newcash
> walk_account_tree: 344, 344, :Assets:Investments:Equities and
> derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade, 0, 0
[...]
> So again, using concatenation instead of format fixes the problem. And
> no GCs have occurred after finishing the loop on the account at issue.

However, here there appears to have been no collections at all (0
before, 0 after)!  So based on this we still cannot rule out a problem
in the GC.

Which makes me wonder: what would happen if you inserted:

        if (account.path == ":Assets:Investments:Equities and derivatives:Taxable:Donald C. Allen 2003 Revocable Trust:TD Ameritrade") 
		GC.collect();

right after the concatenation line to trigger a collection?  Would it
reintroduce the panic?


> And one other tidbit. If I add a precision to %s for the path in the
> format call, like so and change the test for the problem account by
> checking for its guid rather than the truncated path:
[...]
>                 children[i].path = format("%s:%.10s", account.path,
> children[i].name);
[...]
> I get:
> dca at pangloss:~/Software/newcash_d/verifier$ ./verifier
> "--DRT-gcopt=profile:1" /tmp/Finances.newcash
> walk_account_tree: 344, 344, :Assets:Investment:Equities a:Taxable:Donald C.
> :TD Ameritr, 0, 0
[...]
> No panic. So there appears to be a problem in format when dealing with
> strings fed to an unconstrained %s, probably when interacting the
> garbage collector. My guess is that the issue is in format, not the
> gc.

Interestingly enough, again there appears to be no collections done at
all (0 before, 0 after).  So I'm still not sure if we could entirely
rule out the GC at this point.  It would be more interesting to find out
what happens if you insert the GC.collect() line to manually trigger a
collection with the constrained version of the format call.


> That's as far as I've gotten. But I think we know more now than
> before.

This is extremely informative, and thank you for taking the time to do
this in spite of being very busy.  I hope we can eventually ferret out
the cause of the problem, whether it's in the GC, in format, or in your
code somehow.


T

-- 
Error: Keyboard not attached. Press F1 to continue. -- Yoon Ha Lee, CONLANG


More information about the Digitalmars-d mailing list