Stakeholder Axioms

The Good-Enough axiom (p22) summarised the need to provide enough evidence to support decision making with confidence. The value of testing evidence is determined by how confidently stakeholders can make the required judgements.

Of course, confidence cannot be quantified and neither can the value of our testing. This can only be assessed by the people using the evidence produced by or test efforts. To ensure we provide good value, we need to look at how we build confidence in stakeholders.

When a test is run, it generates some evidence of the system working or not working or working in a different way, or it exposes interesting or anomalous behaviour that might require further investigation. If a test has passed, and the system (or its environment) were never to be changed, what value would there be in running that test again? The value of running the same test twice (or more) on an unchanged system or environment is marginal. However, repeated testing is inevitable because tests don’t all pass and systems and environments do change over time.

It makes sense to deliver the most valuable evidence as soon as possible. Our Stakeholders need to receive the most valuable evidence as soon as practical so they can make their decisions sooner and with more confidence. By and large, it is better to correct defects in systems as soon as possible to minimise rework where other deliverables are affected. It is better to run tests that are likely to expose severe defects early to maximise the time available to fix them. The third reason for sequencing tests in a particular order is that the time available for testing may be limited or squeezed. It is sensible to sequence tests so that the most valuable tests are run if execution time is limited, if testing starts late and deadlines are fixed, if testing takes longer than planned or if testing is stopped prematurely. If we don’t sequence tests in this way, it is possible that when testing does stop, we may find that some worthless tests have been run at the expense of more valuable ones.

Our test model makes an implicit, critical, simplifying assumption: that our tests will be run in a known environment. Environments may be delivered late, or not at all, or they are not configured or controlled as required. Inevitably, this will delay testing and/or undermine the stakeholder confidence in any evidence produced.
The general rule must therefore be: Establish the need and requirements for an environment to be used for testing, including a mechanism for managing changes to that environment – as soon as possible.

But it is only when we execute a test and analyse its outcome that we obtain evidence of the actual behaviour of our system. But events blow governments and the best laid test plans off course.

Each test generates a discrete quantum of evidence when run; evidence builds up as we execute more and more tests. We can predict the outcome of these events ahead of time using our oracle but can never know what will happen. The actual outcome of tests is highly dependent on the reliability of the system, the testers, the test environment, test data, our test models and test basis. Unplanned events can stop testing or adversely affect our plans and cause delay to testing and defect fixing or undermine our tests. An agreed incident management process – even if it is informal, will allow the response to the inevitable but unplanned events to be more considered.

In every project, the time arrives where testing stops. If all tests have passed, all defects have been repaired and re-tested and regression tests complete successfully in planned timescales, with all evidence captured – a positive acceptance decision might be a formality. But this happy outcome is a rare event. Usually, the allocated time for testing runs out . There are surely more tests that we could have run; there will probably be defects that have not yet been diagnosed, fixed and re-tested; some defects might not be worth fixing.

The Confidence Axiom: The value of testing is measured by the confidence of stakeholder decision making

Summary Testers must understand the relationship between test evidence and the decisions that stakeholders must make. Testing should focus on providing the evidence that stakeholders require to make decisions with confidence. Consequence if ignored or violated Stakeholders regard the evidence produced by testing as irrelevant, unreliable, incomplete and not aligned with their goals. Questions What...

Read more ...

The Repeat-Test Axiom: Some repeated tests are inevitable

Summary Define and agree a policy for re-testing and regression testing; make an allowance for repeat-tests in estimates and plans. Consequence if ignored or violated Tests to confirm that defect fixes work correctly are not run. Tests to confirm that defect fixes do not cause unwanted side-effects are not run. Questions Under what circumstances will...

Read more ...

The Execution Sequencing Axiom: Run our most valuable tests first – we may not have time to run them later

Summary Sequence tests to ensure the most valuable tests are run if execution time is limited or testing is stopped. Consequence if ignored or violated Stakeholders do not get the evidence they require to make decisions because the required tests have not been executed. Questions What criteria for selection and/or prioritisation will be applied to...

Read more ...

The Environment Axiom: Test execution requires a known, controlled environment

Summary Establish the need and requirements for an environment and test data to be used for testing, including a mechanism for managing changes to that environment – in good time. Consequence if ignored or violated Environments are not available in time or are unsuitable for testing. This will delay testing or cause tests to be...

Read more ...

The Event Axiom: Testing never goes as planned; evidence arrives in discrete quanta

Summary Acknowledge the testing uncertainty principle, and manage stakeholder expectations. Agree a mechanism for managing and communicating events that have a bearing on the successful delivery of test evidence. Consequence if ignored or violated Stakeholders are surprised at the outcomes and progress of testing and blame the testers. Unplanned events stop testing, cause delays, upset...

Read more ...

The Never-Finished Axiom: Testing never finishes; it stops

Summary Recognise that testing is usually time limited and may not complete. Manage the expectations of testers and stakeholders accordingly. Consequence if ignored or violated Testers are frustrated that testing is stopped early. Stakeholders lose confidence and are reluctant to make decisions based on incomplete evidence. Questions Are stakeholders aware that testing probably will not...

Read more ...

Leave a Reply

Your email address will not be published.