Home

evidence

Every test should have value to stakeholders. The tests that we run have value to stakeholders in that they provide evidence to support their acceptance decision making.

The Event Axiom: Testing never goes as planned; evidence arrives in discrete quanta

Summary

Acknowledge the testing uncertainty principle, and manage stakeholder expectations. Agree a mechanism for managing and communicating events that have a bearing on the successful delivery of test evidence.

Consequence if ignored or violated

Stakeholders are surprised at the outcomes and progress of testing and blame the testers. Unplanned events stop testing, cause delays, upset plans and undermine the evidence produced.

Questions

  • How does the test approach accommodate unplanned events?
  • How do plans articulate the uncertainties and expectations of test execution?
  • How will changes to the test bases, system or environment be managed?
  • How will test failures be communicated, tracked and managed?
  • How will unplanned events having an adverse effect on testing be recorded, tracked, analysed and reported?

The Confidence Axiom: The value of testing is measured by the confidence of stakeholder decision making

Summary

Testers must understand the relationship between test evidence and the decisions that stakeholders must make. Testing should focus on providing the evidence that stakeholders require to make decisions with confidence.

Consequence if ignored or violated

Stakeholders regard the evidence produced by testing as irrelevant, unreliable, incomplete and not aligned with their goals.

Questions

  • What evidence do stakeholders need to make decisions with confidence?
  • How will stakeholders use that evidence?
  • How are the goals of testing articulated in plans, specifications, meetings and other communications?
  • How will we ensure that evidence is delivered as early as practical to stakeholders?
  • How will we assure the accuracy and currency of the evidence produced?
  • What preferences for format, detail, frequency, precision, accuracy exist for the evidence produced?
  • How will the evidence be transmitted, acknowledged and reviewed by stakeholders?

The Prioritisation Axiom: Testing needs a mechanism for ordering tests by value

Summary

Testers need to be able to rank tests in order of value and identify which tests are the most valuable.

Consequence if ignored or violated

Stakeholders do not get the evidence they require to make decisions because the necessary tests have not been executed in time.

Questions

  • Who is authorised to define the priorities to be used to select things to test?
  • How will the priorities be articulated?
  • What are the constraints on testing in the current context?
  • Who is authorised to impose, change or remove these constraints?
  • Who will authorise the inclusion or exclusion of tests during scoping?
    Who will authorise the inclusion or exclusion of tests during test design?
  • Who will authorise the inclusion or exclusion of tests during test execution?

The Good-Enough Axiom: The scope of testing and acceptance are always compromises

Summary

Stakeholders and testers must jointly appreciate that there is no limit to testing and that the acceptance decision will always be made on incomplete evidence. In fact, acceptance may occur in spite of evidence, based on information known only to stakeholders.

Consequence if ignored or violated

Stakeholders are frustrated by poor system quality or late delivery because their expectations are unrealistic. Testers are frustrated because they cannot finish testing, the system is imperfect and stakeholders decide to accept regardless.

Questions

  • How much evidence from testing will be required to make the acceptance decision?
  • Who is authorised to make the acceptance decision?
  • What is the mechanism for assessing the value of evidence gathered during testing?
  • What coverage model(s) can be used to judge that enough evidence has been gathered?
  • What criteria will be used to judge that the system under test is acceptable or unacceptable?

The Value Axiom: The value of evidence is for the stakeholder to decide

Summary

The outcome of a test and the way evidence is presented defines its value, regardless of its source.

Consequence if ignored or violated

The approach to testing is an end in itself. The test evidence generated lacks relevance, is ignored, is misunderstood, is inappropriate and has little value to the people that matter.

Questions

  • What acceptance decisions must stakeholders make?
  • What evidence do stakeholders need to make these decisions with confidence?
  • When can the required evidence be gathered?
  • Who needs to provide subject-matter expertise to inform the testing (and make it valuable)?
  • Who are best placed to perform these tests?
  • Are the people or organisations nominated to perform the tests capable of doing so?
  • What environment and infrastructure is required to make the testing meaningful and valuable?

The Stakeholder Axiom: Testing needs stakeholders

Summary

Identify and engage the people or organisations that will use and benefit from the test evidence we are to provide.

Consequence if ignored or violated

There will be no mandate or any authority for testing. Reports of passes, fails or enquiries have no audience.

Questions

  • Who are they?
  • Whose interests do they represent?
  • What evidence do they want?
  • What do they need it for?
  • When do they want it?
  • In what format?
  • How often?
Syndicate content