documented unit tests as examples

[Steven Schveighoffer via Digitalmars-d]

On 5/13/16 5:16 PM, Adam D. Ruppe wrote:
> On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer wrote:
>> Thoughts?
>
> Yeah, I'm not a big fan of this either. A lot of in-comment examples
> have been moved to unittests lately, and I think it is a net negative:
>
> * Running it gives silent output
> * Data representation in source isn't always instructive
> * Assert just kinda looks weird
>
> The one pro would be that it is automatically tested... but is it?
> Consider the following:
>
> ---
> import std.stdio;
>
> ///
> unittest {
>     writeln("Hello, world!");
> }
> ---
>
> That passes the test, but if the user copy/pasted the example, it
> wouldn't actually compile because of the missing import.

This isn't tested, even for DDOC examples. So I'm not super concerned 
about it.

A unit test can use a private module symbol and that won't work in user 
code either.

> Certainly, some surrounding boilerplate is expected much of the time,
> but the unittest doesn't even prove it actually runs with the same
> user-expected surrounding code. It just proves it runs from the
> implementation module: it can use private imports, private functions,
> and more.
>
> So it is a dubious win for automatic testing too.

The idea is to make sure examples actually compile and run in SOME context.

To get down to the lowest level, you can tell someone to copy and paste 
an example in notepad, and run dmd on it, but if they haven't yet 
downloaded dmd, then it won't work ;) It's impossible to test what the 
user is going to do or know before hand.

> D's documented unittests are somewhere in the middle... and I think
> fails to capture the advantages of either extreme.

I'm not knocking documented unit tests. Certainly, without documented 
unit tests, the situation before was that examples were riddled with 
bugs. What I'm pointing out here is that DDOC examples have some 
advantages that unittests cannot currently use.

A potential way to fix this may be marking a unit test as being a 
complete example program that assumes the user has installed proper 
access to the library. Then it won't compile unless you add the correct 
imports, and it's treated as if it were in a separate module (no private 
symbol access). This is probably the closest we can get to simulating a 
user copying an example unit test into his own file and trying to run it.

-Steve