test coverage

Describes how we assess the thoroughness or completeness of our testing with respect to our test model.

The Coverage Axiom: Testing needs a test coverage model or models

Summary

Testers need a means of assessing the thoroughness or completeness of testing with respect to the chosen test models in ways that are meaningful to stakeholders.

Consequence if ignored or violated

Stakeholders do not understand the status of testing. No one knows what has been tested, what has not been tested or if testing is finished.

Questions

  • How will coverage definitions that describe the thoroughness or adequacy of testing be articulated?
  • Can these coverage definitions be used to define a quantifiable coverage measure?
  • How can the coverage measures be related to the goals and concerns of stakeholders?
  • Could these measures support estimation, planning and progress reporting?
  • How will the thoroughness/adequacy of testing be articulated to stakeholders?
  • With respect to the acceptability of the system, what interpretation(s) of these coverage measures could be made?

The Good-Enough Axiom: The scope of testing and acceptance are always compromises

Summary

Stakeholders and testers must jointly appreciate that there is no limit to testing and that the acceptance decision will always be made on incomplete evidence. In fact, acceptance may occur in spite of evidence, based on information known only to stakeholders.

Consequence if ignored or violated

Stakeholders are frustrated by poor system quality or late delivery because their expectations are unrealistic. Testers are frustrated because they cannot finish testing, the system is imperfect and stakeholders decide to accept regardless.

Questions

  • How much evidence from testing will be required to make the acceptance decision?
  • Who is authorised to make the acceptance decision?
  • What is the mechanism for assessing the value of evidence gathered during testing?
  • What coverage model(s) can be used to judge that enough evidence has been gathered?
  • What criteria will be used to judge that the system under test is acceptable or unacceptable?
Syndicate content