Test Run
A test run can comprise the execution of multiple tests. Depending on the choice of command line parameters, a single test, all tests of a specific fixture, all fixtures of a specific module, or all fixtures of a set of modules are run.
Example 1: Running a specific test (SomeTest) which is part of fixture SomeFixture and defined in module somemodule.dll:
cfix32.exe -n SomeFixture.SomeTest somemodule.dll
Example 2: Running all tests of all fixtures found in all modules located in directory c:\foo:
cfix32.exe -r c:\foo
Within a single module, fixtures are run in alphabetic order. Within a fixture, tests are run in the order they have been specified in the CFIX_BEGIN_FIXTURE/ CFIX_END_FIXTURE map with the following additional rules applying:
First, the setup routine (if one has been provided) is run. In case an assertion fails or an unhandled exception is thrown from within the setup routine, the routine is aborted. As the setup routine is considered to play a vital role for successful execution of a fixture, the entire fixture is aborted as well, i.e. no further routines will be invoked.
Next, the test cases of the fixture are run one after another. Running a test case comprises (1) invoking the before routine (if one has been provided), (2) invoking the test routine itself, and (3) invoking the after routine (if one has been provided).
In case of failed assertions or unhandled exceptions, the test case is immediately aborted and execution is resumed at the next test case.
Finally, after all test cases have completed, the teardown routine (if one has been provided) is run. Teardown routines, like setup routines, are optional, so this step does not apply to all fixtures. Again, failed assertions and unhandled exceptions lead to the abortion of the routine.
What exactly does abortion of a routine mean? All failures are first reported so that the user is informed about the situation. By using Windows Structured Exception Handling, cfix then causes the affected routine to be aborted, i.e. no further statements of the routine are executed.
It is notable that the exact behaviour in the event of a failure depends on the context: cfix is aware of the presence of a debugger and adapts its behaviour appropriately. As an example, if a debugger is present and an assertion fails, the test case is not aborted immediately -- instead a brealpoint is issued so that the debugger breaks in on the affected line of code. As soon as the use requests the debugger to resume execution, the test case is aborted.
Moreover, by specifying appropriate command line options, this behaviour can be further adjusted. This includes changing the way cfix deals with unhandled exceptions or specifying that the test run should be aborted as soon as the first failure occurs.
As indicated before, all events -- be it the successful completion of a test case, failed assertions, log messages or unhandled exceptions are reported. Depending on the options chosen by the user, the information is displayed on the console or logged to a text file.
Please note that test cases may spawn additional threads. The next section will cover multithreading in more detail.