"lit"-based test suite

[Johan Engelen via digitalmars-d-ldc]

On Sunday, 3 January 2016 at 18:59:16 UTC, David Nadlinger wrote:
> On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc 
> wrote:
>> - Move the LDC-specific tests from the DMD testsuite into a 
>> few other ./tests/abc directories and run them with lit instead
>
> For the tests that are truly specific to LDC this certainly 
> makes sense.
>
> As fas as the general D tests are concerned that just happened 
> to cause an issue in the LDC glue code but not DMD, I think we 
> should actually try to upstream them to the DMD testsuite, as 
> other compilers will also benefit from the improved coverage. 
> (Not sure whether the powers that be agree with that, though.) 
> For new test cases, we could add them to the lit-based suite 
> first and upstream them every so often.

Agreed.

> How suitable is the lit-based workflow for fail_compilation 
> diagnostic checks? I'm not very familiar with it (yet).

For non-zero return values, there is the "not" LLVM utility:
    not ldc2 compile_error.d
returns zero if ldc2 errored, and returns non-zero if ldc2 
succeeded. There are many tests like that in LLVM/Clang. So then 
I think you'd get something like:
// RUN:  not %ldc2 %s 2&>1 | FileCheck %s
// Test error diagnostic:
// CHECK:  error: bla bla.

> Another question is that of organization. Dumping everything 
> (dmd testsuite runner, general CMake stuff, lit config, 
> individual lit test case files) directly into tests/ might get 
> a bit messy in the long run.

I want to pull lit config, lit runner, cmake stuff out of 
/tests/ir, and put it in /tests. Then all testcases should go 
into a subdirectory of /tests: /tests/codegen, /tests/profile, 
tests/runnable, /tests/whatever.