Skip to content
Topos Uranos
English

Testing

Summary

A program that compiles and prints something reasonable is not, for that reason, tested. This lesson teaches how to test with method: how to derive the cases from the specification, by regions and boundaries; how to gather them into an automated test program that counts the failures and reports them; how to check with assert what cannot happen if the program is correct, and with static_assert, at compile time, what a constexpr function must compute; and how to run the tests with the sanitizers, which uncover the undefined behavior that a correct result hides. Everything rests on one warning: a test can show the presence of an error, never its absence.

Learning objectives

  1. Derive the test cases from the specification of a function: the partitions of its input, the boundaries between them and the special cases, with the result that the postcondition requires in each one.
  2. Write an automated test program that compares the actual results with the expected ones, reports each failure without stopping, and ends with a summary and an exit code that reflects it.
  3. Use assert to check preconditions and invariants, and explain why it disappears when compiling with -DNDEBUG and why its condition must not have side effects.
  4. Use static_assert with constexpr functions to test results at compile time, and run the tests with the sanitizers to uncover the undefined behavior that the results do not reveal.

What testing is

Testing a program means running it with chosen inputs and comparing what it does with what it should have done. Each test case consists of two parts: an input that satisfies the precondition, and the result that the postcondition requires for it, computed beforehand and by another route (by hand, with another formula, with a known table). Without the expected result there is no test, only a run: looking at the output and finding it “reasonable” is an impression, not a comparison.

Now, a function of one int argument admits more than four billion inputs, and one of two arguments about 1.8⋅10191.8 \cdot 10^{19} pairs; no battery of tests goes through them all. If this is so, tests examine a tiny fraction of the possible cases, and from the fact that they all pass it does not follow that the function is correct. Edsger Dijkstra put it this way in 1970: tests can show the presence of errors, never their absence. The distinction is the one that the course Mathematical Logic I draws between checking cases and proving: checking that a formula is true for some values does not prove that it is true for all, whereas a proof (in this course, the invariant of the lesson Designing correct loops) covers all the cases at once. Tests do not replace that reasoning: they find the errors it overlooked.

From this follows the criterion for choosing the cases. If tests cannot confirm correctness, their value lies in finding errors, and a good test case is the one most likely to fail. A case that the author chose because they knew it would work tells nothing. The following program shows this with a function that must return the largest element of a nonempty vector.

C++ program
#include <print>#include <vector>​// Largest element of a list of data.// Precondition: values is not empty.// Postcondition: returns the largest element of values.int largest(const std::vector<int>& values){    int best{0};    for (const int v : values) {        if (v > best) {            best = v;        }    }    return best;}​int main(){    std::println("{}", largest({3, 9, 4}));    std::println("{}", largest({7}));    std::println("{}", largest({1, 2, 3, 4}));    std::println("{}", largest({-5, -2, -8}));}
Output
9740

The first three cases are the ones anyone writes first, and the function passes them: 9, 7 and 4 are correct. The fourth gives 0, a value that is not even in the vector, when it should give −2-2. The error is on line 9: best starts at 0 and not at the first element, so the function is correct only if some element is positive or zero. The typical cases shared that property without anyone choosing it; the fourth violates it on purpose. The fix is int best{values[0]};, which the precondition makes legitimate, because it guarantees that values[0] exists.