Stakeholder Axioms

Testing is a process in which we create mental models of the environment, the system, human nature, and the tests themselves. Test design is the process by which we select, from the infinite number possible, the tests that we believe will be most valuable to us and our stakeholders. Our test model helps us to select tests in a systematic way.

The Test Basis is the source, or sources, of knowledge required to select what aspects of our system we might test and how we might test them.

A Test Oracle is the source, or sources, of knowledge that enable us to predict the outcome of any test. In effect, an oracle tells us what a system does in all circumstances and situations. Implicitly, if an oracle does this, it is perfect. Our oracle might be the same sources as the test basis. Whatever our sources of knowledge are, like test bases, they are fallible.

Coverage is the term we use to describe how we assess the thoroughness or completeness of our testing with respect to our test model. Ideally, our test model should identify coverage items in an objective way. A coverage item is something we want to exercise in our tests. When we have planned or executed tests that cover items identified by our model we can quantify the coverage achieved and, as a proportion of all items on the model, express that coverage as a percentage.

In any non-trivial system, the number of tests that could be run is (effectively) infinite. Compare this huge number of tests to the number of grains of sand on a beach. Our challenge as testers is to identify, from the myriad of possible tests, the ‘pinch’ that are the most valuable and that there is time to execute. We need a way of comparing tests so that one test may be ranked above another so the former is retained and the latter (potentially) discarded.

Our knowledge and the methods we use are partial and imperfect. For example, Test Bases (p30), Test Models (p25), Oracles (p32) and Prioritisation (p38) are all prone to error. It is only sensible to allow for this in our thinking, communications and actions.

Click on the elements in the diagram above or the links below.

Test Model Axiom: Test design is based on models

Summary

Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.

Consequence if ignored or violated

Tests design will be meaningless and not credible to stakeholders.

Read more ...

Test Basis Axiom: Testers need sources of knowledge to select things to test

Summary

Identify and agree the sources of knowledge required to identify what to test. Use multiple sources; compare them and cross-check.

Consequence if ignored or violated

There is no way of knowing what to test. Inconsistencies and gaps in knowledge cannot be identified.

Read more ...

The Oracle Axiom: Testers need sources of knowledge to evaluate actual outcomes or behaviours

Summary

Identify and agree the sources of knowledge required to determine expected outcomes. Use multiple sources; compare them and cross-check.

Consequence if ignored or violated

There is no way of knowing whether a tested system behaves correctly or not. Inconsistencies and gaps in knowledge cannot be identified.

Read more ...

The Coverage Axiom: Testing needs a test coverage model or models

Summary Testers need a means of assessing the thoroughness or completeness of testing with respect to the chosen test models in ways that are meaningful to stakeholders. Consequence if ignored or violated Stakeholders do not understand the status of testing. No one knows what has been tested, what has not been tested or if testing...

Read more ...

The Prioritisation Axiom: Testing needs a mechanism for ordering tests by value

Summary Testers need to be able to rank tests in order of value and identify which tests are the most valuable. Consequence if ignored or violated Stakeholders do not get the evidence they require to make decisions because the necessary tests have not been executed in time. Questions Who is authorised to define the priorities...

Read more ...

The Fallibility Axiom: Our sources of knowledge are fallible and incomplete

Summary Test Bases, Models, Oracles and Prioritisation approach are fallible because the people who define and use them are prone to human error. Consequence if ignored or violated Confidence in the meaning, thoroughness, accuracy and value of tests is misplaced. Questions How have the sources of knowledge required for testing been identified? Who is responsible...

Read more ...

Leave a Reply

Your email address will not be published.