It is very rare to get things right first time. Things don’t pan out as planned, circumstances change and you learn a lot about the best way to do things. This is the joy of pilots – you can test things out and learn as you go before heading into a fully-fledged programme.
We work a lot with partners evaluating pilots. While it may seem premature to evaluate a programme that isn’t yet finalised, we’ve found that our role becomes even more important in these pilot projects. A great example of this is our work with the Premier League on their suicide prevention initiative “Together Against Suicide”. This initiative, in partnership with the Samaritans, set out to raise awareness of the issue of suicide, reduce stigma around it and provide access to support by having mental health and suicide prevention support on matchday fixtures. (For more detail on why football is an effective delivery mechanism for mental health support see our blog here.)
The initiative started as a pilot in 11 Premier League clubs, spread across England, all with very different stadium set–ups, ways of working and community focuses. The Premier League gave the pilot clubs considerable freedom to design their programmes to suit their own settings. Considering that clubs were starting from scratch and may take some time to get up and running, and had a diversity of different approaches to programme design, what value could we bring?
We see our work on this project (and wider pilot evaluations) as falling into three main areas:
1) IDENTIFYING IMPACT
As with most evaluations, the aims and intended impact of a pilot are set out from the start. But unlike established programmes there is less certainty about whether the aims are going to be achieved and how to gather data to be able to tell if they have.
In Together Against Suicide, with such a diversity of clubs and approaches, we knew a very structured evaluation framework wouldn’t work for the pilot as it would be too restrictive and would include too many assumptions that would need to be tested and amended. Instead, we ran alongside the programme, leaving the data collection process relatively flexible, collecting data in different ways at different points across the pilot. This approach allowed us to:
a) validate the initial aims – through our broad data collection approach we were able to evidence that the initiative was having an impact across all its stated aims, raising awareness of the programme across PL fans and the general public, reducing stigma around discussing the topic and seeking help, and increasing access to and uptake of support.
b) explore the unexpected impacts of the programme – by having such an open approach to data collection, we left space for data on other impacts. For example, while the programme had aimed to support fans, our findings showed that a large number of matchday staff had been personally touched by the topic and were supported by the training they received. This element of staff training will continue to be built upon for next season.
2) IDENTIFYING WHAT DATA IS POSSIBLE TO COLLECT AND HOW TO BEST COLLECT IT
A pilot is testing a programme design, but it is also a chance to test out and refine data collection tools and approaches to spot where data may be inconsistent or just not feasible to collect.
A good example of this in the Together Against Suicide initiative is the evolution of how we captured interactions data between fans and support staff. We wanted to understand the number of engagements that fans had with the matchday support team. We started the pilot by asking clubs to feed back what they were counting as engagements or interactions. We soon saw that what clubs counted as an “interaction” varied widely, ranging from an informal chat to a long mental-health support conversation. Through gathering this data we were able to understand the range of different interaction types and create and test a standardised categorisation of interactions that will be used in Year 2 of the programme. This standardisation will allow us to add together interactions across the clubs and report on the number of interactions as a whole across the programme.
3) LEARNING AND SHARING BETWEEN PARTNERS
Pilots have a focus on learning, and using that learning to iterate on all elements of programme design. However, when you have multiple partners, all learning simultaneously, it takes effort and coordination to capture that learning and share it across all partners. That is where we come in.
In the Together Against Suicide initiative, the programme team has already set up quarterly Community of Practice (CoP) sessions for clubs to come together and share experiences and insights. Before each CoP, we analysed our existing data, identifying learnings that could be useful for all clubs, and presented them for discussion at the CoP. An example of this was seeing that some clubs had fixed matchday support hubs (e.g. a stand or a stall), others had support teams roving the stadium, and others were using a combination of both. Having discussed the benefits of all of the models, some clubs adjusted their models – adding roving or fixed elements to their programme design to increase the reach of their support.
What this all means for next year
The initiative has started its second year, expanding to 23 clubs and moving from a pilot to a programme. This next year will build on the foundations established from the pilot: confidence in the aims of the programme, clear data collection processes, insight-informed programme delivery design, and importantly a culture of sharing learning.
So, back to our original question: what value can an evaluation and learning partner bring to a pilot? As Together Against Suicide shows, it is precisely at the pilot stage, when the foundations are being set, that learning is most valuable. Learning doesn’t stop when a pilot becomes a programme, and neither does our role, but it is the aims, data and ways of learning established now that the programme will grow from.
