Examples block

[Solomon E via Digitalmars-d]

On Saturday, 20 August 2016 at 20:39:13 UTC, Engine Machine wrote:
> We have a unittest, what about an examples?
>
> Examples are self contained short programs, each block acts as 
> a "main" function. One can run all the examples and spit out 
> all the output consecutively. It also allows for more robust 
> testing since it adds another layer.
>
> It would provide better examples for docs also. Instead of 
> using assert to assert stuff we can see a real program in 
> action. The output of the example could easily be generated and 
> added to the end of the code.
>
> Seems like a win win! (maybe examples is not a good keyword, 
> but that is moot)

It seems like there could be a library function that 
compile-time-reflects to make a collection of all the functions 
in a module that have names starting with "maintest" and calls 
each of them in a try block, with a catch block that just prints 
the error messages to stderr and incorrect return codes and 
counts the total fails and finally counts the number of tests run 
to return the success ratio.

That would be trivial to write in a language that has runtime 
reflection like Python. Probably someone who knows D well could 
write it for D in a few minutes. It's too hot here (in the 
Seattle area) right now for me with my limited D knowledge to 
feel like trying it instead of kibitzing.

The point would be when writing something that has one 
development-stage 'main', other mains that could be used instead 
or are planned to be used or are just maintests would be live all 
the time (as long as the function that calls them as tests is 
included in the real main or, as it should probably be, in a 
non-release version block or in a unittest block.)

A well-featured interface would allow calling maintests with 
simulated command line args specific for each one, or a list of 
lists of strings times a list of functions, args[][] times 
maintest*[], for more thorough testing. I've used a feature in 
NetBeans for Java (hobby project, not a pro dev here) that calls 
main for a configured build with stored args, but it required 
manual GUI interaction to switch between the builds.

Also a good feature would allow a maintest "success" to be a user 
specified value of a return code for certain args, because when 
writing something that's supposed to return specific error codes, 
that functionality should be tested.

hypothetical usage code:

unittest
{
     auto mainResults = Maintests([1: [["-x", "-y"], ["0"]], 2: 
[["-x", "-A"], ["3"]]);
     // -x and -A switches are incompatible args, to return 3 as 
an arbitrary error code
     assert(mainResults.fails == 1 && 
mainResults.tests["X"][2].success == false);
}

int maintestX (string[] args)
{
     // TODO
     return 0; // it's going to fail the second test, on purpose
}

int maintestY (string[] args)
{
     // TODO
     return 3 * (args[2] == "-A"); // laziest way to pass current 
tests
}

result:

4 tests run with each unittested release build, which would 
otherwise require two additional versions of the module 
containing main to be compiled and run with specific args 2 times 
each, with the correct return values differing for extra 
difficulty, or else require assert(maintestX(["-x", "-y"]) == 0); 
etc. to be written X4 with variations. The number of variations 
to write out manually would increase by the number more test 
cases times the number of variations of main, and wouldn't ensure 
that functions that look like "maintest" are tested.

// ? void maintestZ () { }
// would fail a test that specifies any return value other than 0
// ? int maintest() { throw new Exception("no catch"); }
// would always fail

This might be generalized to
auto results = TestPattern!("maintest*", int[string[]])([["-a"]: 
0]);
which might have some more general uses.

[I wrote the above, then I felt like, nah, I don't want to post 
things that sound like asking for other people to do more work. 
Now that I've done something on it myself, I'm posting the above 
for documentation.]