d2sqlite3 db.run, where lies the bug?

[Ralph Amissah]

ag0aep6g, thank you most of all for fixing the bug, the removal of my immediate
frustration! Also for the thorough step-wise instruction on sleuthing and fixing
it.

>
> If you want, you can make a bug report or a pull request with the fix.
> Otherwise, if you're not up to that, I can make one.
>

Generous, I merely stumbled on it and shouted out, you fixed it. I would be
grateful if you please file the bug and your fix. (I thought I might have more
to report, but your fix has resolved my multitude of issues with several texts).
I did copy my original post to the author of d2sqlite3 (and again this reply
which includes your fix), so he hopefully has the information required to make
the upstream fix. The diff/patch I used:

diff -Naur d2sqlite3-0.16.0/d2sqlite3/source/d2sqlite3/internal/util.d d2sqlite3-0.16.0-1/d2sqlite3/source/d2sqlite3/internal/util.d
--- d2sqlite3-0.16.0/d2sqlite3/source/d2sqlite3/internal/util.d	2018-04-10 20:34:11.584498926 -0400
+++ d2sqlite3-0.16.0-1/d2sqlite3/source/d2sqlite3/internal/util.d	2018-04-10 20:46:09.869812899 -0400
@@ -65,13 +65,14 @@
         {
             import std.algorithm : countUntil;
             import std.string : toStringz;
+            import std.utf: byCodeUnit;

             size_t pos;
             bool complete;
             do
             {
                 auto tail = sql[pos .. $];
-                immutable offset = tail.countUntil(';') + 1;
+                immutable offset = tail.byCodeUnit.countUntil(';') + 1;
                 pos += offset;
                 if (offset == 0)
                     pos = sql.length;

(if patch is contained in patchfile called d2sqlite3_bugfix.diff and at the same
directory level as subdirectory containing d2sqlite3-0.16.0)
cp -av d2sqlite3-0.16.0 d2sqlite3-0.16.0-1 \
&& cd d2sqlite3-0.16.0-1 \
&& patch -Np1 < ../d2sqlite3_bugfix.diff

On Tue, Apr 10 2018, ag0aep6g via Digitalmars-d <digitalmars-d at puremagic.com> wrote:

> On 04/10/2018 08:04 PM, Ralph Amissah wrote:
>> The exact location of problem may be provided in the error statement
>> "core.exception.UnicodeException at src/rt/util/utf.d(292): invalid
>> UTF-8 sequence".
>>
> [...]
>> Mock problem string with test code follows (d2sqlite3 required):
>>
> [... code ...]

[... snip ...]

>
>  From the exception's stack trace we see that
> `d2sqlite3.internal.util.byStatement(immutable(char)[]).ByStatement.findEnd`
> is the deepest non-Phobos function involved. So that's a good first spot
> to look for a bug. Let's check it out.
>
> https://github.com/biozic/d2sqlite3/blob/2e8211946ae0e09646d561aeae1361a695adcc17/source/d2sqlite3/internal/util.d#L64-L83
>
> And indeed, there's a bug in these lines:
>
> ----
> auto tail = sql[pos .. $];
> immutable offset = tail.countUntil(';') + 1;
> pos += offset;
> ----
>
> `pos` is used to slice the string `sql`. That means, `pos` is
> interpreted as a number of UTF-8 code *units*. But then the result of
> `countUntil` is added. `countUntil` counts code *points*. So a number of
> code points is mistaken as a number of code units. That means the next
> slicing can be incorrect and split up a multibyte sequence. And then
> `countUntil` will complain about broken UTF-8.
>
> This can be fixed by letting `countUntil` operate on count code units
> instead:
>
> ----
> import std.utf: byCodeUnit;
> immutable offset = tail.byCodeUnit.countUntil(';') + 1;
> ----
>
> If you want, you can make a bug report or a pull request with the fix.
> Otherwise, if you're not up to that, I can make one.
>
> [...]
>>    - DMD64 D Compiler v2.074.1
>
> That's rather old. I'd recommend updating if possible.

for the time being it reflects the status of the rolling development branch of
Debian. LDC is based on a newer version.

Thanks again.
Ralph Amissah