toString refactor in druntime

[Manu via Digitalmars-d]

On 3 November 2014 19:55, Walter Bright via Digitalmars-d
<digitalmars-d at puremagic.com> wrote:
> On 11/2/2014 11:45 PM, Manu via Digitalmars-d wrote:
>>
>> On 2 November 2014 04:15, Walter Bright via Digitalmars-d
>> <digitalmars-d at puremagic.com> wrote:
>>>
>>> Why would templates make you nervous? They're not C++ templates!
>>
>> What do you mean? How are D templates any different than C++ templates
>> in a practical sense?
>
>
> They're much more straightforward to use, syntactically and semantically.

This isn't anything to do with what I'm talking about. I'm not nervous
because I don't like the D template syntax, it's because I don't feel
it's a good idea for druntime (specifically) to have non-static
interfaces which may expose, or create dependencies on druntime
internals.


>> I want a binary lib to be a binary lib. I don't think it's good form
>> for the lowest level library in the language ecosystem to depend on
>> templates (ie, client-side code generation).
>
>
> What's the problem with that?

Templates which operate on library internals within client code create
more dependencies on the library. It obscures the clarity of the API.


>> This is the fundamental lib that will be present in every D
>> application there is. If it is not a binary lib, then it can't be
>> updated.
>
>
> Have you ever looked at the C openssl.lib? The .h files with it are loaded
> with metaprogramming done with C macros. Yet I've never heard anyone
> complain about it. C .h files for DLLs are typically stuffed with C macros.

I'm not familiar with openssl, but regardless, I wouldn't consider
openssl to lie at the same level in the ecosystem as druntime.
The only library I would use as comparison is the CRT, which is firmly
self-contained, with a well defined API.

I understand your point in principle; if it is effectively 'helper'
code which may be embedded in the client, *but still operates
exclusively via the API*, then I have no issue. It doesn't appear to
me that this is the case here...?


>> Consider performance improvements are made to druntime, which every
>> application should enjoy. If the code is templates, then the old
>> version at time of compiling is embedded into existing client
>> software, the update will have no effect unless the client software is
>> rebuilt.
>> More important, what about security fixes in druntime... imagine a
>> critical security problem in druntime (I bet there's lots!); if we
>> can't update druntime, then *every* D application is an exploit. Very
>> shaky foundation for an ecosystem...
>
>
> The defense presents openssl as Exhibit A!
>
> (The templates really only present the interface to the dll, not the guts of
> it.)

That's fine. But how is that the case here? Performing a string
conversion implies direct access to the data being converted. If the
function is a template, then my app is directly accessing internal
druntime data.


I don't like being classified as 'the offense', in fact, I'm really,
really tired of unfairly being assigned this position whenever I try
and make a stand for things that matter to me and my kind.
I'm just highlighting an issue I recognise in the API, where there is
no way to get a string out of druntime without GC or indirect function
calls.
It seems to be the appropriate time for me to raise this opinion,
since we seem to be in a full-throttle state of supporting @nogc,
which is excellent, but replacing it with indirect function calls
isn't awesome. This takes one problem and replaces it with a different
problem with different characteristics.

I would like to see an overload like the C lib; API receives memory,
writes to it. This may not be the API that most people use, but I
think it should exist.
I then suggested that this API may be crafted in such a way that the
higher-level goals can also be expressed through it. It could be
wrapped in a little thing that may request a memory expansion if the
buffer supplied wasn't big enough:

struct OutputBuffer
{
  char[] buffer;
  bool function(size_t size) extendBuffer; // <- user-supplied
(indirect) function that may expand the buffer in some app-specific
way.
}

toString writes to 'buffer', if it's not big enough, ONLY THEN make an
indirect function call to get more memory.  API is the similar to the
synk callback, but only used in the rare case of overflow.

Perhaps my little example could be re-jigged to support the synk
delegate approach somehow, or perhaps it's just a different overload.

I don't know exactly, other people have much more elaborate needs than
myself. I'm just saying I don't like where this API is headed. It
doesn't satisfy my needs, and I'm super-tired of being vilified for
having that opinion!
I'd be happy with 'toString(char[] outputBuffer)', but I think a
design may exist where all requirements are satisfied, rather than
just dismissing my perspective as niche and annoying, and rolling with
what's easy.

I would propose these criteria:
  * function is not a template exposing druntime internals to the host
application
  * toString should be capable of receiving a pre-allocated(/stack)
buffer and just write to it
  * indirect function calls should only happen only in the rare case
of output buffer overflow, NOT in all cases


>> druntime is a fundamental ecosystem library. It should be properly
>> semantically version-ed, and particularly for security reasons, I
>> think this should be taken very very seriously.
>
>
> openssl!!!
>
> BTW, you should know that if a template is instantiated by the library
> itself, the compiler won't re-instantiate it and insert that in the calling
> code. It'll just call the instantiated binary.

I don't think that's what's on offer here. If toString is a template
(ie, receives an OutputRange, which is a user-defined type), how can
it be that the user's instantiation would already be present in
druntime?
It seems highly unlikely to me. At best, it's unreliable.