Flaw in DIP1000? Returning a Result Struct in DIP1000

[Jonathan M Davis]

On Wednesday, March 21, 2018 17:13:40 Jack Stouffer via Digitalmars-d wrote:
> Consider this example simplified from this PR
> https://github.com/dlang/phobos/pull/6281
>
> ------
> struct GetoptResult
> {
>      Option[] options;
> }
>
> struct Option
> {
>      string optShort;
>      string help;
> }
>
> GetoptResult getopt(T...)(scope T opts) @safe
> {
>      GetoptResult res;
>      auto o = Option(opts[0], opts[1]);
>      res.options ~= o;
>      return res;
> }
>
> void main() @safe
> {
>      bool arg;
>      getopt("arg", "info", &arg);
> }
> ------
>
> $ dmd -dip1000 -run main.d
>
> ------
> main.d(16): Error: scope variable o assigned to non-scope res
> main.d(23): Error: template instance `onlineapp.getopt!(string,
> string, bool*)` error instantiating
> ------
>
> The only way I've found to make the code compile and retain the
> pre-dip1000 behavior is to change the Option construction to
>
> ------
> auto o = Option(opts[0].idup, opts[1].idup);
> ------
>
> How can we return non-scoped result variables constructed from
> scope variables without copies?

The struct being returned would need to be marked with scope (or its members
marked with scope) such that the compiler treated the result as containing
values from the function arguments. I don't know whether that's possible
with DIP 1000 as-is, and that could have some pretty nasty consquences when
you consider how that then limits what can be done with the return value.
Even if we _can_ mark GetoptResult as scope in some manner so that the
copying isn't necessary, you're then just pushing the problem a level up. In
this case, that's probably not a big deal, since this is stuff that's just
going to be used in main and thrown away, but in the general case, having a
struct that won't let you escape any of its members means that you're going
to either need a struct with the same layout but without scope that you can
copy the scope on to, or you're going to need to pull out each of the
members individually to copy them or do whatever you need to do to work
around scope.

My gut reaction is that issues along these lines will either prevent the use
of scope when returning user-defined types as opposed to pointers or dynamic
arrays, and/or they'll force the kind of copying that you're complaining
about. But I don't think that I understand DIP 1000 well enough to know what
it's limitations really are in cases like this. Hopefully, Walter has an
answer.

However, I suspect that we're going to find cases where scope is a blunt
enough instrument that we're going to be forced to either drop it or use
@trusted in places (though in this case, using @trusted would be completely
unreasonable, because you can't guarantee that it _isn't_ a problem for
something from the caller to escape - not without examining the caller code,
which would translate to marking the caller as @trusted, not the function
being called). Actually, I'd be very surprised if we _didn't_ have cases
like that. The question is how frequent those cases are and whether issues
like this are going to pop up enough that it's going to usually make more
sense to simply use pure and manually examine code to mark it as @trusted
rather than use scope so that the compiler can mark a bunch of stuff as
@safe for us. As long as the function doesn't return scope, we're probably
fine, but once it starts returning scope, things start getting interesting.

In this particular case, it may make more sense to just let getopt be @safe
on its own and just let the caller mark all of the uses of & as @trusted. I
know that the bug report that sparked this is trying to make using getopt
completely @safe without requiring the use of @trusted at all, but I don't
think that being forced to allocate memory is worth that, and I don't think
that mucking around with getopt to make it work with ref is worth that.
There are advantages to getopt taking pointers, and I'd rather not see
getopt's API change in a way that breaks code. If getopt itself is @safe,
then it's trivial for the caller to determine that their code is @safe in
spite of the use of & and thus either use @trusted appropriately or just not
bother, since it's in main, which usually isn't doing much fancy. So, I
_really_ don't think that making calling getopt inherently @safe is worth
code breakage if it comes to that, and I don't think that it's worth
allocating memory that we otherwise wouldn't have to allocate.

If it's possible to mark GetoptResult with scope such that we can use scope
without copying, then great, but if it's not, then I'm inclined to argue
that we should just make sure that getopt itself is @safe and not worry
about whether the caller is doing anything @system to call getopt or not.

Regardless, this does raise a potential issue with scope and user-defined
return types, and we should explore how possible it is for DIP 1000 to solve
that problem without forcing copies that wouldn't be necessary in @system
code.

- Jonathan M Davis