Examples block

[rikki cattermole via Digitalmars-d]

On 21/08/2016 9:02 PM, Solomon E wrote:
> On Saturday, 20 August 2016 at 20:39:13 UTC, Engine Machine wrote:
>> We have a unittest, what about an examples?
>>
>> Examples are self contained short programs, each block acts as a
>> "main" function. One can run all the examples and spit out all the
>> output consecutively. It also allows for more robust testing since it
>> adds another layer.
>>
>> It would provide better examples for docs also. Instead of using
>> assert to assert stuff we can see a real program in action. The output
>> of the example could easily be generated and added to the end of the
>> code.
>>
>> Seems like a win win! (maybe examples is not a good keyword, but that
>> is moot)
>
> It seems like there could be a library function that
> compile-time-reflects to make a collection of all the functions in a
> module that have names starting with "maintest" and calls each of them
> in a try block, with a catch block that just prints the error messages
> to stderr and incorrect return codes and counts the total fails and
> finally counts the number of tests run to return the success ratio.
>
> That would be trivial to write in a language that has runtime reflection
> like Python. Probably someone who knows D well could write it for D in a
> few minutes. It's too hot here (in the Seattle area) right now for me
> with my limited D knowledge to feel like trying it instead of kibitzing.
>
> The point would be when writing something that has one development-stage
> 'main', other mains that could be used instead or are planned to be used
> or are just maintests would be live all the time (as long as the
> function that calls them as tests is included in the real main or, as it
> should probably be, in a non-release version block or in a unittest block.)
>
> A well-featured interface would allow calling maintests with simulated
> command line args specific for each one, or a list of lists of strings
> times a list of functions, args[][] times maintest*[], for more thorough
> testing. I've used a feature in NetBeans for Java (hobby project, not a
> pro dev here) that calls main for a configured build with stored args,
> but it required manual GUI interaction to switch between the builds.
>
> Also a good feature would allow a maintest "success" to be a user
> specified value of a return code for certain args, because when writing
> something that's supposed to return specific error codes, that
> functionality should be tested.
>
> hypothetical usage code:
>
> unittest
> {
>     auto mainResults = Maintests([1: [["-x", "-y"], ["0"]], 2: [["-x",
> "-A"], ["3"]]);
>     // -x and -A switches are incompatible args, to return 3 as an
> arbitrary error code
>     assert(mainResults.fails == 1 && mainResults.tests["X"][2].success
> == false);
> }
>
> int maintestX (string[] args)
> {
>     // TODO
>     return 0; // it's going to fail the second test, on purpose
> }
>
> int maintestY (string[] args)
> {
>     // TODO
>     return 3 * (args[2] == "-A"); // laziest way to pass current tests
> }
>
> result:
>
> 4 tests run with each unittested release build, which would otherwise
> require two additional versions of the module containing main to be
> compiled and run with specific args 2 times each, with the correct
> return values differing for extra difficulty, or else require
> assert(maintestX(["-x", "-y"]) == 0); etc. to be written X4 with
> variations. The number of variations to write out manually would
> increase by the number more test cases times the number of variations of
> main, and wouldn't ensure that functions that look like "maintest" are
> tested.
>
> // ? void maintestZ () { }
> // would fail a test that specifies any return value other than 0
> // ? int maintest() { throw new Exception("no catch"); }
> // would always fail
>
> This might be generalized to
> auto results = TestPattern!("maintest*", int[string[]])([["-a"]: 0]);
> which might have some more general uses.
>
> [I wrote the above, then I felt like, nah, I don't want to post things
> that sound like asking for other people to do more work. Now that I've
> done something on it myself, I'm posting the above for documentation.]

Writing your own test runner isn't hard[0]. Although I'm not entirely 
sure this is even on the right continent as for purpose or scope.

Either way while there might be a need here, this is going about it all 
wrong. After all, the documentation we do currently generate is for 
libraries not for whole programs which this is destined for and you're 
gonna wanna do it kinda custom as per the documentation itself.

[0] https://github.com/dlang/druntime/blob/master/src/test_runner.d