PLAIN TEXT - Measuring change support pilot evaluation

Published May 2021.

Written by Laura Fisher, Local Trust.

A version of this paper was published internally by Local Trust in May 2021. This version has been lightly edited to make it accessible for a general audience, with core terms and concepts explained wherever possible.

Studies show that evaluation generally can be a complex process and ​‘communities often lack the resources, skills, time and budget’ to measure and evaluate their own work (Dunkley and Franklin: 2017, p. 114). Acknowledging these challenges, Local Trust has offered support to Big Local areas to better measure the changes they are making in their communities since 2019.

Big Local is a resident-led funding programme run by Local Trust providing people in 150 areas in England with £1.15m each to spend across 10–15 years to create lasting change in their neighbourhoods. The measuring change support sits alongside a broader range of support available for Big Local areas to help them to deliver Big Local. Initial priorities of the Big Local areas usually did not include evaluation as focus was on understanding and delivering the programme. The measuring change support has been developed in response to growing interest from Big Local areas.

The support began in 2019 as a result of growing interest from Big Local areas which has risen dramatically over the past few years as areas make plans for their legacy. Initial priorities of the Big Local areas usually did not include evaluation as their focus was on understanding and delivering the programme. 

The measuring change support offer aims to develop these skills by matching Big Local partnerships with specialist providers to work at a timeframe that suits them. This helps to overcome the resource and time challenges communities face when it comes to evaluating their work. Local Trust also covers the cost of the support, so budget is not an issue. 

The purpose of this offer is to provide a community-led approach to evaluation in order to improve residents’ skills and experience in monitoring, evaluation, and impact measurement. Since its inception, 16 Big Local partnerships have been paired with community-led evaluation specialists to work together to understand the changes they make in their area. A Big Local partnership is a group made up of at least eight people that guides the overall direction of a Big Local area.

The outcomes of the measuring change support are: 

• Increase residents’ skills and capacity to measure the change they are making
• Inform Local Trust on the approaches that are most effective in supporting Big Local areas to measure the change they are making
• Collect more rigorous data and research, for use by both Big Local areas and Local Trust, on the changes they are bringing about
• Support Big Local areas to better work with external organisations to evaluate their activities

Prior to the measuring change support, there was little support to areas around evaluation and impact measurement. The measuring change support came in response to growing interest from areas and also a recognition from Local Trust that this support is necessary to get better information from areas. The terminology of ​‘measuring change’ reflects the light-touch and resident-led ethos of the programme. There are very few reporting requirements on Big Local areas and Local Trust does not expect Big Local areas to undertake an intensive impact evaluation, but we do recognise the value that measurement and evaluation brings to community work. 

In keeping with the ethos of the Big Local programme, this support is demand-led and resident-led, meaning the support can take a variety of forms. Together, the area and support provider, selected from a pool contracted by Local Trust’s research team, will agree and undertake an evaluation plan. Data collection, analysis and reporting is done jointly by the provider and Big Local area, with relevant workshops and training throughout the support period. The support results in multiple outputs, including a report on the data collected, findings and next steps for the Big Local area, as well as any relevant tools, toolkits or frameworks developed through the support which will help the area to continue with measuring change in their community.

From June 2019, 16 areas have been working with 7 support providers. Details on what the areas worked on, and the methods used can be found in Table 1.1 below. Please note this table does not include the 2 areas where the support was ended early.

Table 1.1 – Areas involved in measuring change support pilot:

Table lists ​‘Area’ alongside ​‘Focus’ and ​‘Methods’ as follows:

• Area 1: Focus — Community café, COVID response. Methods — Interviews.
• Area 2: Focus — Digital inclusion. Methods: TBC.
• Area 3: Focus — Community profile; community fridge. Methods: Open data and survey.
• Area 4: Focus — Small grants; COVID response. Methods: Survey.
• Area 5: Focus — COVID recovery plan. Methods: Survey.
• Area 6: Focus — COVID response. Methods: Survey and project feedback.
• Area 7: Focus — Community hub. Methods: Survey.
• Area 8: Focus — Covid response. Methods: Interviews.
• Area 9: Focus — Community hub; community grants. Methods: Survey and interviews.
• Area 10: Focus — Evaluation framework and research tools. Methods: Project feedback.
• Area 11: Focus — Mental health project; evaluation framework. Methods: Peer-led interviews and survey.
• Area 12: Focus — Theory of change. Methods: n/​a.
• Area 13: Focus — Evaluation framework and monitoring toolkit. Methods: Review of existing programme data.
• Area 14: Focus — Theory of change. Methods: n/​a.

Areas began the support following conversations with partnership chairs and Big Local reps about area interest and engagement in evaluation. Big Local reps are individuals appointed by Local Trust to offer tailored support to a Big Local area and share successes, challenges and news with the organisation. Each provider has developed their own approach based on working with other small organisations or community groups and adapts this to suit the style and approach of the Big Local area. The length of support is different for all providers depending on the scope of the work, however all projects were originally anticipated to end by March 2020. Due to various delays, namely the COVID-19 pandemic, support for most of these areas finished around March 2021.

There have also been some iterative changes to the support since its inception in 2019. New providers were added to the pool in January 2020 to provide more varied opportunities for support. A new route of support has also been developed in light of the COVID-19 pandemic which offers areas a less time-intensive and more flexible model than the initial support offer. It offers shorter phases of support, such as theory of change development, without the expectation of a longer commitment but still using the provider model as detailed above.

This evaluation uses the measuring change support pilot to better understand how community-led evaluation work can be supported in Big Local areas. It also focuses on how approaches used can develop communities’ skills and experience in measuring change. This understanding will support Local Trust in developing further measuring change support or other internal support offers and learning can be shared with funders or organisations who may be looking at providing similar support to the communities they work with. 

Research Questions:

• What models and methods of support for community-led evaluation were used?
• What was successful and challenging in these models and approaches?
• Did the support offer increase residents’ skills and capacities? How?
• What are the support needs of communities who want to measure the change they are making?
• How useful are the models in capturing rigorous data and research in community-led programmes?

Partnerships very much guided the onboarding process as they ultimately decided on the support provider and how they wanted to work with them. This process worked well when the partnership had a specific project in mind that they wanted to evaluate and there was a provider that had the relevant experience to work with them. For example, one area approached the research team wanting to evaluate a new digital inclusion project they were working on and there were two providers that had just completed community-led digital inclusion projects. In other cases, however, there wasn’t always a natural fit. This was mainly due to the fact that most partnerships were not clear about what they wanted to use the support for. During initial conversations, people from Big Local areas knew that impact measurement was important, they didn’t have a clear need identified for how the support would fit into their work.

For those who had yet to understand what their need for the support was, the way in which the support was framed during initial meetings between area and provider proved to be key in gaining partnership buy-in. Reflective reports from support providers mention how these meetings were essential for laying the foundations for the rest of the support for many areas. Providers spoke of how they used this meeting, as well as subsequent sessions in the early stages of the support, to ease fears around workload and to reiterate that the support will be flexible and resident-led. What worked well was when providers were able to take an aspect of a partnership’s activities or plan and show how the measuring change support could help with understanding the impact it has, improving its delivery, or promoting its benefits to the community. However, this process of gaining buy-in from the partnership members could often take multiple meetings. 

Observations at initial partnership meetings did show that often partnerships found it a challenge to get to grips with what the measuring change support was and what can be offered. For some the immediate introduction of a support provider may have been overwhelming, particularly for those new to the concept of evaluation. Others misunderstood what their role would be in the support going forward, thinking they were simply gaining a contracted expert to evaluate a project on their behalf with minimal input expected from the partnership. Big Local reps played a vital role at this stage, mainly in encouraging partnerships that the support would be a worthwhile use of their time and acting as a ​‘bridge’ to help partnerships relate the more technical aspects of the support to their work.

However, there were still some areas who were not fully engaged in the support beyond initial meetings, and they went on to face additional hurdles, such as suggesting inappropriate projects for measuring change, including those that were not yet in the pipeline for delivery. This created delays at the crucial early stages of their work, rather than allowing support providers to build on the momentum that can be generated at the start of a new project. Two areas did not finish the support, and in at least one other area work stalled on multiple different workplans due to waning interest from the partnership before the support was eventually completed. Reflections from support providers engaged on these projects suggested that in these areas there was a sense from the partnership that measuring the changes they make was ​‘less urgent’ and so there was little ​‘demand’ from the partnership for the support. From looking at the notes from the initial partnership meetings with those areas and contract notes over the course of the support, it could be argued that these areas never fully understood the role and value of the support and never overcame the hurdles from their initial engagement with measuring change.

While the initial meeting was important to get that buy-in from partnership members to take on the support, chairs and workers proved to be key to maintaining momentum throughout the work. Based on the reflective reports and interviews with partnerships and their workers, it appears that a strong interest in measuring change from the chair, the worker or both is essential to maintaining momentum during the support. As well as helping to frame the support in the early stages of the support, Big Local reps were also credited by support providers as playing a key role in maintaining momentum on the project over time, particularly where Covid19 brought about significant delays. 

In terms of leading on the support, many partnerships successfully created or used existing subgroups to work alongside the support provider which was an effective way to bring other interested partnership members into the project. While not all partnership members will be involved directly in the support this way, the delivery of the support has assumed that partnership members will have varying levels of involvement based on their own personal interests and capacity. What is important is that support providers incorporate time in their plans to update everyone on progress. For many areas, this has involved final, ​‘wrap-up’ sessions with all partnership members to summarise the work carried out and discuss what this might mean for the partnership going forward, whereas others have kept the partnership updated throughout with frequent agenda items at partnership meetings. Furthermore, where a smaller group are involved in leading on the work with the support provider, it is important that this group is as resident-led as possible, in keeping with the principles of the support. In some instances, the support provider has had to be fairly directive to the partnership about who should be involved at different stages of the support in order to prevent involvement solely from non-residents or workers. 

Finally, it is also important for the momentum of the support that there is interest in measuring change, and the support worked best when providers could rely on the input of both the chair and a worker. Where the chair, worker or partnership are not on the same page about the support, can exacerbate existing conflict and cause delays. 

‘A project worker was keen on monitoring and evaluation, but their enthusiasm was not shared by the partnership board…This person largely took on this role for themselves; when they left, this enthusiasm for measuring change went with them.’ – support provider.

The relationships between the provider and the partnerships is a key factor in driving the support forwards, and it could be seen in those areas where there were personnel changes, either at the provider organisation or in the partnership, how time needed to be taken to rebuild connections between the two parties before work could progress.

Across all models, the additional capacity brought in by the provider is of clear benefit to partnerships. Not only is the technical support around developing a theory of change or tools and frameworks for evaluation valuable, but the support partnerships get through the reflective process of measuring change was also seen to be a major benefit, as well as having someone to codify those reflections into processes or tools that areas can adopt and use going forward. One chair also spoke about the benefits of bringing in an external provider who was seen by the partnership to be a neutral figure. This was beneficial in some of the more challenging aspects of the work, such as when encouraging the partnership to think about what their aims were for individual projects and how they linked, or didn’t, with the overall goals for their Big Local. This demonstrates the importance of providers being skilled at facilitation and understanding the resident-led context, alongside the more technical knowledge of evaluation and research. 

From the provider perspective, one provider reflected on how working directly with the community enabled them to gain a deeper understanding of the issues through discussing them with residents themselves. This was seen to aid the support by creating a wider understanding of the community context. They reflected that the process felt ​‘less remote’ than when working with charities where there is often minimal stakeholder engagement and the conversations take place through the lens of the professional charity workers. However, another provider reflected on the complexity and challenges of working with a community group where the ​‘level of engagement with the work was low in comparison’ to other groups they have worked with in the past. Assessing what is different for providers when working directly with communities through the process of measuring change, and how this might lead to better outcomes or support for partnerships could be a topic for further exploration as more areas complete the support.

Each Big Local followed a different process throughout the support, despite similar approaches being used. This highlights the value of flexibility in the delivery of measuring change support, and its role in ensuring partnerships felt ownership over the support by being able to have a say in the direction it takes. This was credited by those interviewed as being one of the most beneficial aspects of the process. 

As projects in Big Local areas changed and new projects arose, support providers had to adjust existing workplans to adapt to the new context. This was even more important due to the dramatic changes to delivery caused by the COVID-19 pandemic. This flexibility was highly valued by the partnerships we spoke to, and academic research notes how the consideration of changing contexts in community work is important in building ownership over evaluation and ensuring that an evaluation plan is developed ​‘that communities feel able and willing to implement’ (Dunkley, et al., 2017). Frequent communication was also cited as important throughout, and those interviewed from Big Local partnerships did recognise that it was the contact from providers during the first 6 to 9 months of the pandemic that ensured the projects were completed and not delayed further. 

‘Once Covid was fully flowing, it was certainly the case that [the provider] was chasing us to come back to the project. It did fit with our priorities at the time but we just weren’t thinking of it in that way –Big Local chair.

While the support aims to be resident-led there is still space for the providers to suggest or adapt their approaches based on the partnership’s needs. Some providers were good at developing proposed workplans which suggested how their time could be used, whereas others left this entirely up to the areas to decide. While the latter approach is in line with the support being resident-led, this may have been too much pressure for partnerships to decide for themselves at an early stage in the support. This was particularly the case for partnerships that were very new to the idea of evaluation, or where there were still reservations about the support as a whole. This caused delays in the initial stages of the support and it took a great deal longer to finalise and start a workplan. Where providers were able to guide or suggest ways forward for partnerships, both in the initial stages of support and throughout the process, project momentum was maintained, even if workplans had to be adapted as time went on. 

Relatedly, despite the flexibility built into the support, this did not produce much variety in the models or approaches used. It is clear that work may need to be done in the future to identify alternative methods, such as action research or participatory appraisal, that could work in individual areas, as well as supporting areas to see these as options for their work with providers. These may feel too ambitious for partnerships to consider if they are new to the concept of measuring change or may be felt to be ​‘the next step’ for partnerships, following initial work with providers to understand what changes they are trying to make in their areas. Alongside encouraging providers with specific specialisms to discuss a variety of approaches in initial meetings with partnerships in the future, it is hoped that the move to more flexible support offers will also allow Local Trust to identify opportunities in areas where these alternative approaches may work.

Many partnerships were interested in the support in order to better articulate the changes they’re making to others in their community. But over the period of the work with their provider, areas begin to recognise the links between the support and their legacy. Legacy planning is something that Big Local partnerships are beginning to spend more time thinking about, as they approach the end of the programme they want to ensure they leave something behind in their community. Aligning the support with the development or implementation of a new Big Local plan, can be an effective way to make those links clearer. 

‘[During the planning process, the provider] supported us with a sort of strategic thread throughout and from that everybody now understand what it is we’re trying to do over the next three years of the plan and what changes we hope to see.’ – Big Local chair.

However, linking the two pieces of work is not always straightforward. There may be capacity challenges for partnerships during the plan development phase, and this can reduce interest in any measuring change work that happens alongside it. Due to a pre-existing relationship between the support provider and the plan development consultant, one area managed this through bringing together the two support partners. This was ​‘ultimately beneficial’ for the partnership as the sharing of information meant the measuring change provider was able to ​‘integrate the work we were doing on impact measurement into the plan…this helped the partnership members to grasp the strategic importance of measuring change in a way that might not have been possible if they had not simultaneously been working on their plan.’ 

An objective of the measuring change support is that areas will feel able to continue with monitoring and evaluation after the support has ended. It is clear that models of support that led to the development of evaluation frameworks or resources to support with monitoring and data collection have embedded in partnerships the sense that evaluation is a process that will extend beyond the lifetime of the measuring change support. Not only does this understanding begin to frame partnership’s conversations about their legacy, it also supports broader strategic thinking across all that partnerships do. 

‘If you’ve got a prospective new project and you look at our new framework and measuring tools and think ​‘well, we’re not going to do any of that from this project’, then now you can say ​‘why are we doing this project?’ Is it a project that actually fits with what we as a Big Local want to achieve?’ – Big Local worker.

Chairs reported better strategic thinking across the partnership after the support, with a greater awareness of how projects and activities should fit in with the aims for the area. They spoke of how project development would be easier going forward now there were clear frameworks in place that would enable alignment between individual projects and the intended aims.

From conversations with key stakeholders in partnerships and the reflections of the support providers, there has been definite learning on partnerships around evaluation. Partnerships now have a greater understanding of measurement and feel they now understand both what is measurable and how to measure more challenging outcomes such as soft skills and people’s thoughts and feelings. 

‘We’ve got community grants projects and we’ve been able to take what we worked on with [the provider] and add wellbeing questions to the paperwork such as ​‘how are you feeling this week?’ And now we will track wellbeing over time.’ – Big Local worker.

Similarly, there has been a change to how some partnerships perceive evaluation, with a recognition that evaluation is not just something ​‘that big funders do’ and is achievable and appropriate for Big Local partnerships too. This links to the improved ability of partnerships to think strategically about their work as mentioned above, with partnership’s feeling their increased understanding of the concept of evaluation will help them to better hold others to account, such as workers or external delivery partners. 

Alongside a better understanding of measurement, partnerships have also developed a greater awareness of the processes that sit alongside a successful evaluation framework, including data collection, data storage and the need to start considering project evaluation from project conception. This also links to future plan development and legacy planning, with partnership’s recognising the benefits of having a bank of useful, thematic data, analysis and learning to draw on. At the start of the measuring change support, many areas identified how they had a lot of data that had been collected over the years, but had done very little with it. Partnership members interviewed spoke of how they will no longer be in this position as they now understand why they collect data and what they can do with it. 

It is clear that confidence as a partnership has also increased. Those interviewed spoke of how those in the partnership now felt they had the confidence to speak about how the work they do contributes to bigger aims, or how to request appropriate monitoring information from external organisations that they fund. 

‘As a partnership we know that the projects we work on are good because we live and breathe them, other people outside say ​‘oh yeah, and?’, so now we’ve learned how to give them the ​‘and’’ – Big Local chair.

However, where any skill development and confidence building has taken place, this has occurred mainly for chairs, workers and those partnership members who were involved most closely in the support via subgroups. This aligns with a similar finding from the review of Local Trust’s support offer, which found that the ​‘project lead’ who provided the link between the support provider and the partnership often gained the most in terms of skills and development (Local Trust, 2017). Yet for measuring change there is the expectation that eventually all partnership members will gain the skills and confidence that those who were more closely involved have, as new processes and practices are embedded and partnerships begin to see the benefits of evaluation across their work. 

‘When you can show [partnership members] the impact of projects when they complete, hopefully that will show them that it is actually worth doing this little bit of extra work that shows the benefits at the end’ – Big Local chair.

For one Big Local area, the increased knowledge and skills has already begun to move beyond the partnership itself. They have shared their framework and evaluation toolkit developed through the support with external organisations they work with to deliver projects, some of which are small, hyperlocal, and grassroots. Not only has this sharing of knowledge benefited the local voluntary community sector, but it has improved the reputation of the Big Local too. They now feel they are better able to communicate with stakeholders and manage relationships that previously may have fallen apart due to issues with monitoring, and the chair feels their reputation has improved because of their ability to better communicate the changes they want to achieve and speak more confidently about how they will measure this. 

‘There’s been a lot of learning around how we maintain the communication, build the relationship, make sure the partner understands from the very outset what we’re trying to achieve and therefore is completely bought in with the measuring we would like them to do’ – Big Local chair.

Most partnerships were not able to test new tools or engage in data collection with the community as they may have initially planned due to COVID-19 and restrictions on social interactions. One area did receive a report evaluating their community café and others undertook some small research pilots in the few months of support prior to the pandemic, but these were not able to be as in-depth as expected or progress into bigger research projects. However it has to be noted that as some areas were not able to test new tools or try research techniques, there is a potential risk that when partnerships do attempt in-person use of certain tools, they may not work as well in practice , or they may not feel confident enough to use them without further support in place. Additional support may be needed for some partnerships to address this. As restrictions lift and partnerships begin delivering activities and events in person, future measuring change support should be able to include data collection. 

‘Due to the Covid restrictions we were not able to use all the evaluation tools and techniques that we had originally envisaged using, and therefore the skills and knowledge that we have been able to leave with the group are limited. Further training or support of evaluation methodology would help to overcome this.’ – support provider.

While the research undertaken during the measuring change support may not be fully rigorous at this stage, what the introduction of new processes and frameworks has done is show the partnerships that they are able to capture data over time, and they are aware of the tools and skills to be able to do this. People involved in the support believe that the skills and knowledge developed through the support will enable them to improve their practices and delivery in their area, through the development of future plans or planning for their legacy (Dunkley et al., 2017).

It will be important to return to this question on rigor in the data and research generated through measuring change support in the future, when COVID-19 rules allow partnerships to undertake data collection and analysis or specific research projects alongside their support provider.

To conclude, it is clear that the measuring change bespoke support has been of value to those Big Local areas who participated in the pilot, through skill and confidence development, the provision of additional capacity and by building links between measurement, strategic thinking and legacy planning. The flexibility that is built into the support has been essential to maintaining area buy-in across the project, particularly given the delays caused by the Covid-19 pandemic. Yet there is also evidence that it is important for providers to guide the partnership through key decisions in the support, in order to maintain momentum, and working closely with chairs, workers and reps can be beneficial. This guiding approach is also necessary during the initial introductions to measuring change, where partnership buy-in hinges on the support being clearly linked to the current or planned activities of the partnership, as well as drawing attention to the other benefits of the support as detailed throughout this paper.

It is also important to note that despite the challenges of the Covid-19 pandemic and the delays this caused, areas demonstrated their commitment to evaluation and the measuring change support by continuing the work throughout a difficult time. This is meaningful as community-led evaluation can be challenging in and of itself and will always require a significant amount of time and capacity. The planned, and actual, lengths of the measuring change support contracts is evidence of this.

Looking ahead, further work is required to ensure that skill and capacity development moves beyond a select few in the community. Yet, while we cannot expect all involved in Big Local partnerships to engage in impact measurement, the measuring change bespoke support does provide the flexibility to allow the support to be as community-led as possible. It is hoped that this will only be further developed as face-to-face delivery becomes possible which will lead to even greater benefits for Big Local areas involved in the support.

Drawing on the above findings, this piece has identified some of the key support needs for communities who want to measure the changes they are making in their local area. Some of these may also be relevant to other Local Trust support offers, or instances of community-led evaluation in the wider voluntary sector. 

These key support needs for communities who want to measure the changes they are making in their local area include:

• supporting the community to identify their own need
• flexibility
• additional capacity and knowledge
• improvement of skills and capacities
• links to broader strategic thinking
• embedding of processes
• support to interpret data and share findings.

For partnerships to be engaged with the support, it is incredibly important that they understand the purpose of the evaluation and how it can benefit them. Focussing specifically on what areas will gain out of participating in the measuring change support and putting forward a clear explanation of the offer is essential in ensuring that areas understand and buy-in to the support from the beginning of the process. Going forward, the above findings on how the support has improved strategic thinking in partnerships, links to area’s legacy planning and plan development, and development of resident’s skills and capacities, will help with communicating to partnerships what the support can offer them. Drawing on the internal review of Local Trust’s support offer, as well as feedback from support providers in the pilot, more work has also been done by Local Trust to ensure that Big Local reps are clear on what the support can offer to areas so that they can play an informed role in framing the support for Big Local areas (Local Trust, 2018). Additional work may also be necessary to help areas that may not have an identified need for evaluation, especially during the assessment process. Areas that may struggle with strategic thinking could benefit greatly from measuring change support, but there needs to be a more thorough way to help them identify their own evaluation support need so they recognise the benefits of the support.

The flexibility of the measuring change model has been key to maintaining momentum and partnership interest in the work over time. Local Trust have already introduced a more flexible support offer that allows partnerships to take up an aspect of support that may fit better with their context and expectations. One area has already completed one portion of this new flexible support offer and has now decided to move on to a further phase of support with their support provider. This suggests that the more flexible model allows for a natural progression through a measuring change journey, rather than expecting an up-front, longer-term commitment from partnerships. 

As mentioned above, it was clear that many Big Local partnerships had already collected data on the work they had done, but were unsure on what to do with it or how to link it to the broader aims and outcomes of their work. Measuring change support not only brings the knowledge on how to do this, but also the capacity to facilitate the reflections of the partnership on this data and develop resources from it. Conversations with areas showed that the process of creating frameworks and tools to measure their impact had felt like a worthwhile use of their time but was something they could not have undertaken alone. The partnership that had shared its framework and tools with local organisations it works closely with also reported similar feedback from these groups: that they would never have had the capacity to consider their evaluation processes and the best methods for capturing data. This shows that the value of bringing in additional capacity can extend beyond the Big Local partnership and into the local voluntary community sector, with particular benefits to hyperlocal, grassroots groups who face the same challenges around ​‘resources, skills, time and budget’ (Dunkley et al., 2017). 

The improvement of resident’s skills and capacities in relation to impact measurement is a cornerstone of the measuring change bespoke support. While the evaluation showed that, initially, learning and development resides with a select few within most partnerships, workers and chairs spoke about how they planned to embed and expand this learning to those who were not directly involved in the support. Alongside the more obvious learning around specific methods and approaches to measuring impact, partnerships also spoke of their greater understanding around the processes and implementation of evaluation, and the generation of evaluation frameworks had clearly provided them with a structure to understand their work and inform their decisions in the future. Unfortunately, the Covid-19 pandemic has provided a barrier to the improvement of skills and capacities, with many partnerships not able to try out and test their new tools and learning or develop their experiences as researchers. As the support is able to become more experiential when restrictions lift, we will be able to understand in more depth the ways in which the support offer helps communities measure the changes they make in their area.

The links between evaluation and planning strategically for the future are evident, however, for some of the Big Local areas involved in the measuring change pilot, the development of this skill has been a positive, but unintended consequence. Through the development of the evaluation frameworks and theories of change, areas have unlocked ​‘key questions’ they can ask themselves of the work they deliver, commission or contract. Partnerships have reported how this has allowed for more nuance in their decision-making and has clarified the link between what they do on the ground and their overall aims and goals for their Big Local area. This has begun to contribute to Big Local’s thoughts on their post-2026 legacy, and it will be interesting to track this effect over time as legacy plans take shape. The development of a Big Local plan also proved to be a natural channel for shaping how Big Local partnership progressed from thinking about the changes they make in their area to thinking about what they ultimately want to achieve and how they can evidence that. While working on a plan at the same time as taking on measuring change support can prove a challenge, it may be useful to explore how these two things can better align in more detail through future support. Indeed, Big Local partnerships have already begun to approach Local Trust to take up measuring change support with the aim of it contributing to plan development or enhancing a newly approved plan.

With new frameworks and tools in place, partnerships who have completed the support are now in the position of ensuring these become an everyday part of the work they do on Big Local. Having capable workers will definitely help partnerships with some aspects of this, but where accountability needs to sit with partnership members, or where workers are lacking the necessary skills, work may need to be done to ensure the use of certain resources and frameworks becomes habitual. Similarly, partnerships clearly link the embedding of processes to ensuring that all partnership members are interested in and understand the value of measuring change. One provider identified how there was a risk of processes not being fully embedded by those who were not involved in their development, and more work may be needed so those individuals can see the relevance to their role. Another partnership is attempting to demonstrate the value of new processes and their new framework to all partnership members by nominating project champions which will encourage board members, with some worker support, to utilise the tools and processes to have oversight of projects. This is an area that Local Trust will want to examine further in the future, alongside the extent to which partnerships find it easy to embed what they did during the support over a period of time. 

Where partnerships are able to use their new frameworks to gather data on the work that they do, there will come a time when that data can be looked at holistically to get an understanding of the changes they are making in their area. We know that a lot of partnerships were interested in the measuring change support in order to better articulate the changes they are making to others in their community. As mentioned, one partnership received a research report around their community café, and this has gone on to be used in an external funding bid, showing that as partnership attentions turn to legacy, they may also wish to demonstrate changes in their community to funders or local authorities. It is understandable that when partnerships want to create outputs with new data, such as reports, infographics, or even just interpreting their findings to assess internally, this may feel challenging or time-consuming to those unfamiliar with the process.. As restrictions lift and in-person data collection feels more conceivable, it would be interesting to look at how future waves of support are able to build knowledge and skills around data collection, data analysis and the presenting of findings to key stakeholder groups. 

Dunkley, R.A. and Franklin, A. (2017) ​‘Failing better: The stochastic art of evaluating community-led environmental programs’, Evaluation and Program Planning, 60, pp. 112–122.

Local Trust (2018) ​‘Review of the support offer’, Local Trust (no longer available).

To better understand how community-led research methods have worked in Big Local areas, Local Trust collected and analysed primarily qualitative data from the areas and providers involved in the bespoke support. The below appendix details the data that contributed to the evaluation. 

Observational data was collected at partnership meetings when the providers are first introduced to the partnership. This data looks at how the partnership engage with the providers and their understanding of measuring change at the start of the process, providing a comparison point with the reflections from providers following the support (see below). 

Throughout the course of the measuring change support, providers have attended sessions with Local Trust to share learning and experiences of their work on the programme. This feedback contributed to the reflective questions that providers completed at their end of their projects (see below). A session in July 2020 was used to gather information on the effects of the COVID-19 pandemic on their work. 

Progress on the support was monitored through frequent conversations between providers and Local Trust, either via Zoom, telephone, or email. the project management website Monday was used to maintain regular updates on contracts over time. 

Zoom interviews were conducted with up to 2 people from 3 Big Local areas. These interviews were held with both a worker and the chair of the partnership in each area. Questions asked during the interview included:

1. As part of your area’s measuring change support, you worked with [support provider]. What support did they provide?
2. How and why did your partnership select the themes for the work with [support provider]?
3. What helped things progress with the support? Would you have done anything differently?
4. What do you feel you have learned personally from the measuring change support? How did you feel the support helped you understand the changes you’re making?
5. What do you feel your partnership has learned from the measuring change support?
6. How will you continue to use what you have learnt through measuring change in your Big Local area? How will you use the outputs produced? How has it helped you with creating your new plan/​thinking about your next plan [delete as applicable]
7. Do you have any final reflections or comments about the measuring change support?

As a final output of the support, providers were contracted to deliver a report covering work conducted during the support. The reports were intended to inform the Big Local area and Local Trust on the findings of the support, which should an include an evaluation of a project or priority. Providers were given a flexible structure to use as a guide when producing their report. 

Finally, all providers wrote a report for to Local Trust that detailed their reflections on the effectiveness of the support, what worked well or less well and potential support needs going forwards. The following report questions were developed by Local Trust alongside the support providers:

1. What helped your delivery of the support? What hindered it?
2. If you were to redo this work with the Big Local area, would you have done anything differently?
3. Is working with Big Locals different to how you’ve worked with other community groups? If working with Big Locals is different, how did you adapt the delivery of the support to best suit them?
4. How did you manage limited capacity in the area?
5. Who in the area took on key roles in the delivery of this support? How was this decided?
6. How did you get buy-in from the partnership to deliver the support? How did you maintain it?
7. How will the area continue to measure change? How equipped is the area to do this after the support ends (based on their skills and understanding of measuring change)?
8. Is there any support you think they could benefit from in the future?

As highlighted in the main report, some aspects of the evaluation of the measuring change support pilot were not able to be interrogated due to the effects of the Covid-19 pandemic. Further questions or areas to consider have also been raised throughout the pilot but insufficient information was available at this stage to investigate them further. They are captured here for future reference or future evaluations of the measuring change support.

• What are the best approaches to frame the measuring change support to interested Big Local areas?
• As more areas directly approach Local Trust about receiving measuring change support, how does this impact their ownership of the project, their experience of the support and what they learn from it?
• How can we encourage partnerships to undertake more complex or unfamiliar approaches?
• If areas were to undertake more complex or unfamiliar research approaches through the measuring change support (such as action research or participatory appraisal), how might this affect their experience of the support or what they take from it?
• How are areas conversations or plans around their legacy impacted by their increased capacity and understanding around measuring change?
• How will experiences of measuring change support alter as support is done in-person and the learning process can become more experiential?
• What else can be drawn out about the differences for providers in working directly with communities to measure change through the support?
• How well do areas involved in this initial wave of measuring change support embed the new processes and practices they have developed?
• How well do areas involved in this initial wave of measuring change support promote the understanding of measuring change and improve the skills of their wider partnership and beyond?
• How can measuring change support help with developing skills around direct data collection or first-hand research, data analysis, and presenting of findings?
• How do areas use the information they have gathered through the measuring change process and beyond (such as improved decision-making, leveraging for match funding)?