value

About Test Axioms

Paul Gerrard introduced the idea of Test Axioms in posts on his blog in the spring of 2008. Over a few months their definitions evolved and in May 2008 he summarised the thinking behind them and tabulated 16 proposed axioms. Further evolution has occurred and with some changes, they have been used in the Tester's Pocketbook.

There was quite a reaction to the proposed axioms. Some people rejected the idea, saying there were no such things. Others were more supportive and offered new axioms or alternate definitions.

Paul believes that there are a set of rules or principles that provide a framework for all testing. But there is no single agreed definition of test. This is mostly because every consultant and author has tended to write their own definition to suit their own purposes (and Paul admits to being as guilty as the rest of them). As an industry, we are hamstrung because of this. Our clients are confused, and we get distracted by discussions on definitions because of not-invented-here mentalities and our competitive instincts.

An axiom is something believed to be true, but cannot be proven in any practical way. It could be disproven by experiment or experience and we should be prepared to be proven wrong and welcome attempts to do this.

But some people object to the notion of Test Axioms and say that nothing in testing is certain. There are no axioms. All testing rules, principles, techniques, approaches etc. are heuristic. Heuristics have value in some contexts, but are limited in application, usefulness, accuracy etc. in other contexts. They are limited or fallible in known ways.

Here is a different way of looking at axioms then. The axioms have been defined in a way that testers can, for all practical purposes, regard them as axiomatic. If anyone devises a testing context where the axioms are violated, we need to think again: Perhaps the axiom should be scrapped or changed or its scope of applicability defined.

So far, Paul has not received any concrete examples that invalidate the Test Axioms as stated in the Pocketbook.

The Test Axioms are an attempt to provide a context-neutral set of rules for testing that identify the critical thinking processes and motivations for all test approaches.

The Fallibility Axiom: Our sources of knowledge are fallible and incomplete

Summary

Test Bases, Models, Oracles and Prioritisation approach are fallible because the people who define and use them are prone to human error.

Consequence if ignored or violated

Confidence in the meaning, thoroughness, accuracy and value of tests is misplaced.

Questions

  • How have the sources of knowledge required for testing been identified?
  • Who is responsible for the content in these sources?
  • Has the content of the sources been agreed unanimously?
  • Have these sources stabilised?
  • Have these sources been verified against other references or validated against the experience of contributors and other stakeholders?
  • What measures can be taken to minimise the impact of error in our sources of knowledge?
  • If sources are fallible or conflict, who or what is the final arbiter?

The Prioritisation Axiom: Testing needs a mechanism for ordering tests by value

Summary

Testers need to be able to rank tests in order of value and identify which tests are the most valuable.

Consequence if ignored or violated

Stakeholders do not get the evidence they require to make decisions because the necessary tests have not been executed in time.

Questions

  • Who is authorised to define the priorities to be used to select things to test?
  • How will the priorities be articulated?
  • What are the constraints on testing in the current context?
  • Who is authorised to impose, change or remove these constraints?
  • Who will authorise the inclusion or exclusion of tests during scoping?
    Who will authorise the inclusion or exclusion of tests during test design?
  • Who will authorise the inclusion or exclusion of tests during test execution?

The Good-Enough Axiom: The scope of testing and acceptance are always compromises

Summary

Stakeholders and testers must jointly appreciate that there is no limit to testing and that the acceptance decision will always be made on incomplete evidence. In fact, acceptance may occur in spite of evidence, based on information known only to stakeholders.

Consequence if ignored or violated

Stakeholders are frustrated by poor system quality or late delivery because their expectations are unrealistic. Testers are frustrated because they cannot finish testing, the system is imperfect and stakeholders decide to accept regardless.

Questions

  • How much evidence from testing will be required to make the acceptance decision?
  • Who is authorised to make the acceptance decision?
  • What is the mechanism for assessing the value of evidence gathered during testing?
  • What coverage model(s) can be used to judge that enough evidence has been gathered?
  • What criteria will be used to judge that the system under test is acceptable or unacceptable?

The Value Axiom: The value of evidence is for the stakeholder to decide

Summary

The outcome of a test and the way evidence is presented defines its value, regardless of its source.

Consequence if ignored or violated

The approach to testing is an end in itself. The test evidence generated lacks relevance, is ignored, is misunderstood, is inappropriate and has little value to the people that matter.

Questions

  • What acceptance decisions must stakeholders make?
  • What evidence do stakeholders need to make these decisions with confidence?
  • When can the required evidence be gathered?
  • Who needs to provide subject-matter expertise to inform the testing (and make it valuable)?
  • Who are best placed to perform these tests?
  • Are the people or organisations nominated to perform the tests capable of doing so?
  • What environment and infrastructure is required to make the testing meaningful and valuable?
Syndicate content