Expanding Tests

Things have been going well. You have run multiple PDSA cycles and found several interventions that appear to have made beneficial changes. This section explores what can be done to consolidate your early successes and further expand your ability to learn about improving your system.

Iterative Testing

One of the great benefits of the PDSA cycle is the ability to carry out iterative tests of change. Each cycle generates new information and raises new questions. Subsequent PDSA cycles can be used to answer these questions and build upon your latest knowledge. Successive cycles enable you to adjust your changes, increase the size of your subsequent tests and increase your degree of belief in the changes you are making. As you grow in knowledge, confidence and scale, so increases the acceptance of people making your interventions more likely to become adopted. So, what should you change first?

Read more from the Institute for Healthcare Improvement on linking tests of change.

Degrees of Belief

Initially our degree of belief in a change will be limited by the lack of knowledge to support our test and predictions. As you gain a deeper understanding through testing, your degree of belief will increase. You degree of belief changes as you act on your PDSA cycles and increase the scale or scope of subsequent tests. Confidence is boosted by testing in different conditions, with different people, at different times of day, in different locations and by trialling changes that don’t work. This is important because the degree of belief you hold will influence the changes you are willing to make. Below are characteristics of low and high degree belief states.

Low degree of belief

New Idea
Test requires a workflow change
Colleagues are resistant to change

High degree of belief

Same changes worked elsewhere in a similar setting
Previous testing in your location has been successful

A word of caution; individuals at all stages of the PDSA cycle tend to overestimate the likelihood of success.

Determining Test Size

You should always begin with a small test. This allows for observation of change with minimalization of risk. Ultimately, the size of your test is a function of your degree of belief in change creating improvement balanced by the consequences of failure. Sizes can be determined by creating a matrix exploring this balance. NHS Improvement has produced a guide on bullet-proofing which helps you explore potential consequences of your intervention.

Low Belief

Minor Consequences

Medium

Low Belief

Major Consequences

Very Small

High Belief

Minor Consequences

Small to medium

High Belief

Major Consequences

One test cycle prior to implementation

Tests which could have major consequences if not successful are often characterised by the introduction of a new service, technology or procedure. They may also involve expensive equipment or have a direct impact on patients.

A good rule of thumb is to test in units. Begin with the smallest size of one provider, one patient and one encounter. If this worked as expected, you can then increase the scale or scope of subsequent tests. You should increase either scope or scale in a test, not both. This makes interpretation of results easier. Scale and scope are defined below.

Scale

Scale is the timespan or number of events included in a test cycle. It can be increased by including more patients, events or time. Before scaling you need to make sure that the current change being tested works for the current test group. This is because if it doesn’t work for the current group, why would it work for a larger group!

If your change appears to be effective, you can begin to think about whether or not the results would be reproducible in different situations. Appreciate that as your intervention is applied to other situations, the specific circumstances will influence whether or not the intervention works there as well. Thus, a larger test will result in the need for further testing as additional problems are highlighted.

You can get a head start on this by considering what a full-scale implementation would look like. Map backwards and identify the various levels of growth that need to occur to get you from your first small test to your desired larger test size. Contemplate the various difficulties you might encounter at each stage and prepare for them.

Now you’ve decided to increase the scale, how much should you increase it by? A good idea is to increase it 5-fold from the previous test. This is a manageable increase and has the added benefit of highlighting logistical issues that need solving before a larger role out.

Scope

Scope describes the variety of conditions under which a test occurs (different patients, different times, different staff). It is important to explore a variety of test conditions over several cycles to demonstrate the potential for success in a range of conditions. This will increase confidence that change will lead to improvement. Consider changing the type of patients, the staff involved or the time of the test.

For tests that involve patients, try out the change on staff before using it with patients. This allows identification of problems before patients are involved and confirms the validity of the test to your staff, as well as training them in how it will be applied.

Simultaneous Testing

If you are familiar with increasing the scale and scope of your tests, you may want to consider increasing the speed of improvement by running multiple tests at the same time. This is possible using the planned experimentation method. This method allows you to explore multiple conditions in parallel with different teams responsible for different cycles. You use different combinations of tests to recognise relationships that work. Planned experimentation requires significant forethought and planning and is beyond the scope of this article. If you do decide to use it as a strategy, you should try to reduce interaction between the teams otherwise it may become difficult to tell the difference between each test. Remember, you want your PDSA cycles to build upon each other, not replicate one another! Further information can be found in the book “Quality Improvement Through Planned Experimentation”.

Read more from the Institute for Healthcare Improvement on testing multiple changes at once.