silly is released - new test runner for the D programming language

[Bogdan Szabo]

Nice work Anton!

It's nice to see that people are interested in writing better 
test runners. This project reminds me of `tested`: 
http://code.dlang.org/packages/tested


On Monday, 13 August 2018 at 17:32:32 UTC, Anton Fediushin wrote:
> On Monday, 13 August 2018 at 11:57:34 UTC, Atila Neves wrote:
>> On Monday, 13 August 2018 at 04:13:46 UTC, Anton Fediushin 
>> wrote:
>>> On Sunday, 12 August 2018 at 21:33:21 UTC, Dechcaudron wrote:
>>>> On Sunday, 12 August 2018 at 15:07:04 UTC, Anton Fediushin 
>>>> wrote:
>>
>>> Problem with unit-threaded and similar tools is that they are 
>>> too complicated for no particular reason.
>>
>> Yes, I often wake up and think to myself "what feature can I 
>> add that aren't going to be useful to anyone at all?" then 
>> write some code.
>>
>> Nobody adds complication "for no particular reason". Even 
>> people who shouldn't be doing it do it because they're trying 
>> to accomplish something else.
>>

Yes... and the main problem is that usually the features added 
fit the author vision and that vision it's hard to fit all the 
project.

>>> Hacking into dub.json to add some scripting into it is not 
>>> something everybody wants to waste their time on.
>>
>> Agreed. Other than manually listing modules however (which is 
>> what I do these days anyway due to a dub bug), I don't know of 
>> a better way.
>>
>> I wish there were a compile-time trait like 
>> __traits(allModules). I've thought of writing a DSI
>
> Sadly, there's no better way yet. Silly imports `dub_test_root` 
> which contains all of the modules. That's why no `main()` 
> should be defined and tests must be run with `dub test

embedding the runner code in a project is tricky for compiled 
programming language... maybe that's why javascript and ruby have 
a good selection of test runners. I tried to hide the mess in the 
runner subpackage... I'm not proud of it, but it works.

>
>>
>>> Another thing, these tools are trying to be everything people 
>>> might need adding all kinds of features nobody really uses. 
>>> For example, assertions in unit-threaded and a lot of 
>>> different reporters in trial.
>>
>> This is a good point. I think I should split unit-threaded up 
>> into separate dub subpackages. Thanks for the idea!
>
> That'd be nice.
>

The reporters from trial are optional, and you can tell the 
runner to use whatever you want. I insisted on the reporters part 
because I think this is an important feature of a test runner. I 
can't think of how you would manage thousands of tests without 
good reporting.

>>
>>> These tools also advertise usage of built-in `unittest` 
>>> blocks for integration testing. I think it's just wrong 
>>> because `unittest`s are obviously meant for unit testing and 
>>> slapping integration tests on with some duct tape and zip 
>>> ties is not a good solution. Integration testing needs it's 
>>> own tool and it's quite possible that I'll end up writing one 
>>> soon or later.
>>
>> I'm curious as to why you think that's the case.
>
> Well, 'advertise' might not be the right word for that, but 
> both unit-threaded and trial provide means to write integration 
> tests. Even though there is support for some simple things like 
> creating temporary files and defining steps, it goes beyond 
> "unit testing" but is not enough for integration tests. This 
> kind of testing can require things like setting up a database 
> or running other services which cannot be done with existing 
> tools.
>
>> ...

For me, integration tests are long and sometimes are too many. 
When you embed the integration tests inside the source code makes 
harder to find the tested code.

I know that we lack of features for the integration tests, but my 
time is limited and I would like work on other projects too...

I would like to see if we can make those runners compatible, in 
the sense that if someone wants to switch from silly to 
unit-threaded or trial, to be done with no headaches.