Home

delivery axioms

The Never-Finished Axiom: Testing never finishes; it stops

Summary

Recognise that testing is usually time limited and may not complete. Manage the expectations of testers and stakeholders accordingly.

Consequence if ignored or violated

Testers are frustrated that testing is stopped early. Stakeholders lose confidence and are reluctant to make decisions based on incomplete evidence.

Questions

  • Are stakeholders aware that testing probably will not complete in planned timescales?
  • Do testers appreciate that all tests might not be run, that the acceptance decision may be made on incomplete evidence and that the accepted system could be imperfect?
  • Are testers prepared to inform, advise and facilitate the acceptance decision?
  • What role will the testers have in presenting test evidence and completion (or incompletion) status?
  • How will the testers support stakeholders in judging the relevance/significance of inconclusive evidence?

The Event Axiom: Testing never goes as planned; evidence arrives in discrete quanta

Summary

Acknowledge the testing uncertainty principle, and manage stakeholder expectations. Agree a mechanism for managing and communicating events that have a bearing on the successful delivery of test evidence.

Consequence if ignored or violated

Stakeholders are surprised at the outcomes and progress of testing and blame the testers. Unplanned events stop testing, cause delays, upset plans and undermine the evidence produced.

Questions

  • How does the test approach accommodate unplanned events?
  • How do plans articulate the uncertainties and expectations of test execution?
  • How will changes to the test bases, system or environment be managed?
  • How will test failures be communicated, tracked and managed?
  • How will unplanned events having an adverse effect on testing be recorded, tracked, analysed and reported?

The Environment Axiom: Test execution requires a known, controlled environment

Summary

Establish the need and requirements for an environment and test data to be used for testing, including a mechanism for managing changes to that environment – in good time.

Consequence if ignored or violated

Environments are not available in time or are unsuitable for testing. This will delay testing or cause tests to be run in the wrong environment and undermine the credibility of evidence produced.

Questions

  • Who is responsible for the acquisition, configuration and support of test environments?

The Execution Sequencing Axiom: Run our most valuable tests first – we may not have time to run them later

Summary

Sequence tests to ensure the most valuable tests are run if execution time is limited or testing is stopped.

Consequence if ignored or violated

Stakeholders do not get the evidence they require to make decisions because the required tests have not been executed.

Questions

  • What criteria for selection and/or prioritisation will be applied to sequence the planned tests?
  • What criteria for selection and/or prioritisation will be applied to sequence unplanned or ad-hoc tests?
  • What are the constraints to sequencing tests?
  • How will dependencies between tests be minimised and managed?
  • Under what circumstances can tests run out of sequence?

The Repeat-Test Axiom: Some repeated tests are inevitable

Summary

Define and agree a policy for re-testing and regression testing; make an allowance for repeat-tests in estimates and plans.

Consequence if ignored or violated

Tests to confirm that defect fixes work correctly are not run. Tests to confirm that defect fixes do not cause unwanted side-effects are not run.

Questions

  • Under what circumstances will failed tests be re-run?
  • Under what circumstances will passed tests be re-run?
  • What criteria will be applied for the retention of tests for reuse?
  • Under what circumstances will retained tests be discarded or amended?
  • For the purpose of planning, how will the following be estimated or defined:
    • The proportion of tests that fail?
    • The time required for defect fixing and re-testing?
    • The proportion of tests to be used for regression purposes?

The Confidence Axiom: The value of testing is measured by the confidence of stakeholder decision making

Summary

Testers must understand the relationship between test evidence and the decisions that stakeholders must make. Testing should focus on providing the evidence that stakeholders require to make decisions with confidence.

Consequence if ignored or violated

Stakeholders regard the evidence produced by testing as irrelevant, unreliable, incomplete and not aligned with their goals.

Questions

  • What evidence do stakeholders need to make decisions with confidence?
  • How will stakeholders use that evidence?
  • How are the goals of testing articulated in plans, specifications, meetings and other communications?
  • How will we ensure that evidence is delivered as early as practical to stakeholders?
  • How will we assure the accuracy and currency of the evidence produced?
  • What preferences for format, detail, frequency, precision, accuracy exist for the evidence produced?
  • How will the evidence be transmitted, acknowledged and reviewed by stakeholders?

Delivery Axioms

Stakeholder Axioms

Syndicate content