[Issue 18026] Stack overflow in ddmd/dtemplate.d:6241, TemplateInstance::needsCodegen()

[d-bugmail at puremagic.com]

https://issues.dlang.org/show_bug.cgi?id=18026

Mike Franklin <slavo5150 at yahoo.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |slavo5150 at yahoo.com

--- Comment #16 from Mike Franklin <slavo5150 at yahoo.com> ---
I've been looking into fixing this.  The problem is this block of code: 
https://github.com/dlang/dmd/blob/750f0245c2882887cdb830c171e9221719e0da28/src/dmd/dtemplate.d#L6291-L6320

---
if (!minst)
{
    // If this is a speculative instantiation,
    // 1. do codegen if ancestors really needs codegen.
    // 2. become non-speculative if siblings are not speculative

    TemplateInstance tnext = this.tnext;
    TemplateInstance tinst = this.tinst;
    // At first, disconnect chain first to prevent infinite recursion.
    this.tnext = null;
    this.tinst = null;

    // Determine necessity of tinst before tnext.
    if (tinst && tinst.needsCodegen())
    {
        minst = tinst.minst; // cache result
        assert(minst);
        assert(minst.isRoot() || minst.rootImports());
        return true;
    }

    // Issue 18026 - Stack overerflow due to recursive call to needsCodegen
    if (tnext && (tnext.needsCodegen() || tnext.minst))
    {
        minst = tnext.minst; // cache result
        assert(minst);
        return minst.isRoot() || minst.rootImports();
    }

    // Elide codegen because this is really speculative.
    return false;
}
---

Specifically, the stack overflow occurs at...
---
if (tnext && (tnext.needsCodegen() || tnext.minst))
{
    minst = tnext.minst; // cache result
    assert(minst);
    return minst.isRoot() || minst.rootImports();
}
---
... due to the recursive call to `needsCodegen`

The whole function `needsCodegen` looks like a mess.  There's probably a way to
implement the algorithm as a loop instead of recursive function calls, but I'm
not yet sure what to do to fix this.

--