test design

The process of selecting things to test from a test model.

The Fallibility Axiom: Our sources of knowledge are fallible and incomplete

Summary

Test Bases, Models, Oracles and Prioritisation approach are fallible because the people who define and use them are prone to human error.

Consequence if ignored or violated

Confidence in the meaning, thoroughness, accuracy and value of tests is misplaced.

Questions

  • How have the sources of knowledge required for testing been identified?
  • Who is responsible for the content in these sources?
  • Has the content of the sources been agreed unanimously?
  • Have these sources stabilised?
  • Have these sources been verified against other references or validated against the experience of contributors and other stakeholders?
  • What measures can be taken to minimise the impact of error in our sources of knowledge?
  • If sources are fallible or conflict, who or what is the final arbiter?

The Prioritisation Axiom: Testing needs a mechanism for ordering tests by value

Summary

Testers need to be able to rank tests in order of value and identify which tests are the most valuable.

Consequence if ignored or violated

Stakeholders do not get the evidence they require to make decisions because the necessary tests have not been executed in time.

Questions

  • Who is authorised to define the priorities to be used to select things to test?
  • How will the priorities be articulated?
  • What are the constraints on testing in the current context?
  • Who is authorised to impose, change or remove these constraints?
  • Who will authorise the inclusion or exclusion of tests during scoping?
    Who will authorise the inclusion or exclusion of tests during test design?
  • Who will authorise the inclusion or exclusion of tests during test execution?

The Coverage Axiom: Testing needs a test coverage model or models

Summary

Testers need a means of assessing the thoroughness or completeness of testing with respect to the chosen test models in ways that are meaningful to stakeholders.

Consequence if ignored or violated

Stakeholders do not understand the status of testing. No one knows what has been tested, what has not been tested or if testing is finished.

Questions

  • How will coverage definitions that describe the thoroughness or adequacy of testing be articulated?
  • Can these coverage definitions be used to define a quantifiable coverage measure?
  • How can the coverage measures be related to the goals and concerns of stakeholders?
  • Could these measures support estimation, planning and progress reporting?
  • How will the thoroughness/adequacy of testing be articulated to stakeholders?
  • With respect to the acceptability of the system, what interpretation(s) of these coverage measures could be made?

The Oracle Axiom: Testers need sources of knowledge to evaluate actual outcomes or behaviours

Summary

Identify and agree the sources of knowledge required to determine expected outcomes. Use multiple sources; compare them and cross-check.

Consequence if ignored or violated

There is no way of knowing whether a tested system behaves correctly or not. Inconsistencies and gaps in knowledge cannot be identified.

Test Basis Axiom: Testers need sources of knowledge to select things to test

Summary

Identify and agree the sources of knowledge required to identify what to test. Use multiple sources; compare them and cross-check.

Consequence if ignored or violated

There is no way of knowing what to test. Inconsistencies and gaps in knowledge cannot be identified.

Test Model Axiom: Test design is based on models

Summary

Choose test models to derive tests that are meaningful to stakeholders. Recognise the models’ limitations and the assumptions that the models make.

Consequence if ignored or violated

Tests design will be meaningless and not credible to stakeholders.

Syndicate content