Either I'm confused or the gc is

[donallen]

On Saturday, 24 October 2020 at 15:04:53 UTC, Steven 
Schveighoffer wrote:
> On 10/24/20 10:21 AM, donallen wrote:
>> On Friday, 23 October 2020 at 23:41:05 UTC, rikki cattermole 
>> wrote:
>>> Thanks!
>>>
>>> The problem is not in that block of code I believe.
>>>
>>> Now how about bind_text?
>>>
>>> Is that still using c variadic args?
>> 
>> No. Read on.
>> 
>> The detective in me couldn't resist (I really need a Peugeot 
>> and a basset hound) and so I did some more work on this 
>> despite not really having the time and I have made some 
>> progress.
>
> Very nice!
>
> I wonder if the corruption is happening on the account.guid. 
> Since we are talking about sqlite, which is a local DB, and not 
> sending information to a server (where it can't be corrupted), 
> it's possible the guid is being corrupted causing the premature 
> exit of the loop.
>
> Can you print out the guid after in the premature exit case to 
> make sure it hasn't changed?

A good thought, but it doesn't change:

         // Now do the children of this account
         // First determine how many there are
         string guid_before = account.guid;
         bind_text(count_children, 1, account.guid);
         int n_children = one_row(count_children, 
&get_int).integer_value;
         if (n_children > 0)
         {
             size_t collections_before = 0;
             size_t collections_after = 0;
             Account[] children = new Account[n_children];
             string guid_after = account.guid;
             bind_text(find_children, 1, account.guid);
             int i = 0;
             while (next_row_available_p(find_children, 
&reset_stmt))
             {
                 children[i].name = 
fromStringz(sqlite3_column_text(find_children, 0)).dup;
                 collections_before += 
GC.profileStats().numCollections;
                 //children[i].path = (account.path ~ ":" ~ 
children[i].name);
                 children[i].path = format("%s:%s", account.path, 
children[i].name);
                 collections_after += 
GC.profileStats().numCollections;
               children[i].guid = 
fromStringz(sqlite3_column_text(find_children, 1)).dup; // guid
                 children[i].commodity_guid = 
fromStringz(sqlite3_column_text(find_children, 2)).dup; // 
commodity_guid
                 children[i].flags = 
sqlite3_column_int(find_children,
                         3) | account.flags & 
(account_flag_descendents_are_assets
                         | 
account_flag_descendents_are_liabilities | 
account_flag_descendents_are_income
                         | account_flag_descendents_are_expenses | 
account_flag_descendents_are_marketable
                         | 
account_flag_self_and_descendents_are_tax_related
                         | 
account_flag_descendents_need_commodity_link); // flags
                 i = i + 1;
             }
             if (i != n_children)
                 panic(format("walk_account_tree: bad child index, 
%d, %d, %s, %s, %s, %d, %d", i, n_children, account.path, 
guid_before, guid_after, collections_before, collections_after));
             if (account.guid == 
"c369abf6ec2c7222e3fdd174ce2c0c9a") {
                 writeln(format("walk_account_tree: %d, %d, %s, 
%d, %d", i, n_children, account.path, collections_before, 
collections_after));
             }
             foreach (child; children) {
                 walk_account_tree(child, ancestor_flags | 
account.flags);
             }


dca at pangloss:~/Software/newcash_d/verifier$ ./verifier 
"--DRT-gcopt=profile:1" /tmp/Finances.newcash
walk_account_tree: bad child index, 231, 344, 
:Assets:Investments:Equities and derivatives:Taxable:Donald C. 
Allen 2003 Revocable Trust:TD Ameritrade, 
c369abf6ec2c7222e3fdd174ce2c0c9a, 
c369abf6ec2c7222e3fdd174ce2c0c9a, 33, 34

>
>> This does not panic! In theory, this should consume the same 
>> amount of GC memory as the format call (though I have not 
>> looked at the format code; there may well
>> be intermediate allocations as it does its work).
>
> Just for informational purposes, an append statement between N 
> strings allocates one block that holds all of them. Format 
> appends, so it may involve multiple allocations, though it's 
> possible it does a similar shortcut. I'm very interested in 
> this code, I will look at the case for format, and see if I can 
> sleuth out any possible corruption possibilities.
>
>> 
>> No panic. So there appears to be a problem in format when 
>> dealing with strings fed to an unconstrained %s, probably when 
>> interacting the garbage collector. My guess is that the issue 
>> is
>> in format, not the gc.
>
> Yeah, I tend to agree. More specifically, it could be in 
> appender as well (which is what format uses to allocate memory).
>
>> That's as far as I've gotten. But I think we know more now 
>> than before.
>
> Good work, and I think we can probably get to the bottom of it, 
> lots of great information!
>
> -Steve