Test Design Axioms

Stakeholder Axioms

Testing is a process in which we create mental models of the environment, the system, human nature, and the tests themselves. Test design is the process by which we select, from the infinite number possible, the tests that we believe will be most valuable to us and our stakeholders. Our test model helps us to select tests in a systematic way.

The Test Basis is the source, or sources, of knowledge required to select what aspects of our system we might test and how we might test them.

A Test Oracle is the source, or sources, of knowledge that enable us to predict the outcome of any test. In effect, an oracle tells us what a system does in all circumstances and situations. Implicitly, if an oracle does this, it is perfect. Our oracle might be the same sources as the test basis. Whatever our sources of knowledge are, like test bases, they are fallible.

Coverage is the term we use to describe how we assess the thoroughness or completeness of our testing with respect to our test model. Ideally, our test model should identify coverage items in an objective way. A coverage item is something we want to exercise in our tests. When we have planned or executed tests that cover items identified by our model we can quantify the coverage achieved and, as a proportion of all items on the model, express that coverage as a percentage.

In any non-trivial system, the number of tests that could be run is (effectively) infinite. Compare this huge number of tests to the number of grains of sand on a beach. Our challenge as testers is to identify, from the myriad of possible tests, the ‘pinch’ that are the most valuable and that there is time to execute. We need a way of comparing tests so that one test may be ranked above another so the former is retained and the latter (potentially) discarded.

Our knowledge and the methods we use are partial and imperfect. For example, Test Bases (p30), Test Models (p25), Oracles (p32) and Prioritisation (p38) are all prone to error. It is only sensible to allow for this in our thinking, communications and actions.

Click on the elements in the diagram above or the links below.