The Oracle Axiom: Testers need sources of knowledge to evaluate actual outcomes or behaviours

Summary

Identify and agree the sources of knowledge required to determine expected outcomes. Use multiple sources; compare them and cross-check.

Consequence if ignored or violated

There is no way of knowing whether a tested system behaves correctly or not. Inconsistencies and gaps in knowledge cannot be identified.

Questions

  • What sources of knowledge should be used to derive expected outcomes to validate observed behaviour?
  • Do stakeholders approve the use of these sources as a test oracle?
  • What confidence can we have in these sources?
  • Do expected outcomes need to be documented prior to test execution?
  • What is the derivation/heritage/provenance and reliability of these sources?
  • What are the levels of authority or precedence of these sources?
  • In case of dispute, who or what source will be the final arbiter?