Cookies on Knowhow

We use cookies in order for parts of NCVO Knowhow to work properly, and also to collect information about how you use the site. We use this information to improve the site and tailor our services to you. For more, see our page on privacy and data protection.

OK

Skip to content. | Skip to navigation

Community-made content which you can improve Case study from our community

Developing and using a theory of change

This page is free to all
How NCVO developed its theory of change, and what we did with it next.

Background

NCVO is the membership organisation for the voluntary sector. Our ultimate goal is for voluntary organisations and volunteers to make a bigger difference. To achieve this, we seek to improve the operating environment for voluntary organisations, strengthen voluntary organisations, grow and enhance volunteering wherever it takes place, and connect people and organisations.

This case study describes how we developed our theory of change and then what we did with it once we’d developed it.

The issues we faced

At NCVO we have been working on improving how we measure the difference that our work makes, as we wrote about in this blog post. Some individual projects and teams were already using theory of change to describe their work and direct what information they collected for evaluation. We realised that an organisation-wide theory of change would help to keep the planning for our work joined up and focused on the difference that we make. We face particular challenges in articulating our value because our outcomes are so far from the front-line impact. We recognised that a theory of change would help us to communicate this work to others.

There were also gaps in the data that we collected for evaluation and the ways we reported on that data. We wanted reports to be more useful to trustees and staff, and to drive decision making, so we could truly say that we were evidence-informed.

The actions we took

Developing our theory of change

First we developed our theory of change. We brought teams together from across NCVO to articulate the links between what we do and the difference that we make. We already had a five-year strategy, so we organised workshops around each of our strategic aims. Before the workshops we held drop-in sessions for staff so that they understood the difference between impact, outcomes and outputs.

Each workshop was attended by around 20 people and facilitated by a consultant from our own NCVO Charities Evaluation Services team. We started with draft theories of change for each aim which we debated in the workshop – an activity that involved robust discussion and lots of post-it notes! We then pulled together the content from all the workshops into a master map, which we displayed in the office so everyone could comment. After a lot of further discussion, we had an agreed copy. We then employed a professional designer to get it ready to share on our website.

We’ve written more about the process of developing our theory of change and what we learned in this blog post.

Using our theory of change for planning

We’ve made our theory of change the foundation of our planning process. Our planning sessions are focused around the end outcomes of the theory, for example ‘voluntary organisations are more sustainably resourced’. At each session, we discuss how we can provide services or support that will help contribute to the end outcome we’re focusing on.

Using our theory of change for monitoring and evaluation

We then needed to work out what we were going to measure to help us to evidence the difference that we make. The theory of change was a helpful starting point, but needed to be translated into key pieces of information that we could track to help us understand whether we were making a difference – and, if so, how.

In our 2017/18 organisational planning process we prioritised key outcomes from the theory that we would collect evidence against. The theory has over 40 outcomes and we agreed that starting with a smaller number for measurement was a good idea.

We continued to work with our own NCVO Charities Evaluation Services team to turn the theory into a monitoring and evaluation framework – a plan for data collection. Building on the success of our collaborative approach to developing the theory, we ran participatory workshops and had individual and group discussions with teams across NCVO to help develop indicators they could report on for each priority outcome.

Next, we brought together everything that we had discussed with different teams and created a monitoring and evaluation framework. This framework specified what information needed to be collected for each priority outcome, who was responsible, how it would be collected, how often and how it would be reported.

We also spoke with and surveyed trustees about what information they wanted to receive and how they wanted to receive it. Based on this, we designed a reporting template for trustees.

We started using the framework and template for reporting in the second quarter of our 2017/18 financial year and have been refining and improving it as we go.

Positive outcomes

The theory of change process had really great engagement from across the organisation and we are hugely appreciative of the time and enthusiasm that staff put in. We kept the process flexible so that staff could engage by taking part in the workshops or making individual comments.

The theory of change has helped colleagues to communicate NCVO’s work to external stakeholders. Not only has it helped to have it on our website, it’s also been a useful tool for colleagues who speak in public about NCVO’s work. It helps them to link our work to our outcomes and to benefits for our members and the wider voluntary sector.

Since we made the theory central to our planning process, it’s become much more joined up and it’s helping to build links across different teams. Outcomes-focused planning required cross-team working. Through this, teams learned more about each other’s work and how colleagues viewed their own work. For example, the NCVO Charities Evaluation Services team learned that their work on communicating the value of the sector was seen as a high priority by colleagues, which they had not expected.

The new monitoring and evaluation framework helps us to have more of an outcomes focus in our reports to trustees. By linking our reporting to the outcomes in the theory we can begin to evidence how our work makes a difference.

The framework has also helped to bring together work done by different teams. Sometimes the work done by more than one team feeds into the same outcome, and we’ve been able to see that much more clearly. This has enabled teams to understand more about each other’s work and collaborate more easily.

It was invaluable to have the NCVO Charities Evaluation Services team’s support and expertise in getting the theory of change, framework and report template set up. They also supported those who were interested, from various teams, to develop their skills in evaluation and reporting. As a result of this, colleagues in other teams were able to take the lead on refining and updating the framework for the 2018/19 financial year to reflect our updated planning priorities.

Negative outcomes

In this section we’re focusing on the aspects of the process that were more difficult. These aren’t negative outcomes as such, but more areas where we know we have some further development to do.

We struggled to include our internal-facing work in the theory. Our HR, finance, facilities, fundraising and IT work is vital to NCVO’s sustainability and enables us to deliver the external-facing work. But we didn’t initially know how to do it justice without introducing a lot of complexity to the theory. We also have good systems for measuring the success of our internal-facing work, so it was felt that the priority was to focus on representing the external-facing first. Since the first theory has been developed, a small group came together to develop a separate internal-facing one as well.

A lot of our work is very difficult to evaluate as our outcomes are quite far removed from the ultimate impact on individuals and communities served by voluntary organisations. This is particularly true of some of our policy and campaigning work. Evaluating some of this work will take a big investment of time and resources. We listened carefully to staff in those teams to understand what could and could not be measured as a routine part of people’s work. For now, we have accepted that there are gaps in the information that we can collect, and in the future we will invest in evaluating the outcomes of specific areas of our work in order to be able to make better strategic decisions.

It also took some time to get to a cost-effective, user-friendly reporting format for teams to put their data into. Initially we had a single spreadsheet that teams could edit, but most found that unwieldy and difficult. We’ve now developed individual forms for each team. It means that they can’t see the whole picture the way they used to, but it was more important to have something that was easy to use.

We’re also not quite there in terms of using the data to make better decisions. This is partly because we cannot measure everything that we would want to, but it’s also about SMT, staff and trustees engaging with the data more. As time goes on and the quality and consistency of data improves, this should become more embedded. Eventually, we’d like to create a dashboard to make reporting more engaging and transparent.

Lessons learnt

  • Ideally a theory of change and strategy development process would go together. We weren’t able to do that this time but when the strategy comes up for review we will integrate it with the theory of change.
  • Expect a theory of change and monitoring and evaluation framework to take time to develop and need more than one iteration. It can be frustrating but it’s better to go slow and get it right eventually than rush it.
  • It’s worth being as participatory as possible when developing a theory of change so that you draw on everyone’s expertise. Sometimes that might mean being a bit less precious about wording and format as long as everyone agrees and understands.
  • Theories of change are meant to be used. Integrating it into our planning process and other processes has been helpful in keeping it alive.
  • Teams may only have limited capacity and resource to engage in a collaborative process. Use their time as wisely as possible and be responsive to their concerns and queries. It’s important that people are happy with the indicators that they’re reporting against and feel they’re a fair reflection of their work.
  • Remember to keep feeding back insights from the data to the teams who provided it. People are much more motivated to collect and provide information if they can see how it’s being used.
  • Monitoring and evaluation does need some investment in order to be of real value. People gave up their time willingly to contribute to the framework and report on the data, and we’re now considering where we might invest to evaluate strategic priority areas.

You can find out more about how to develop a theory of change and how to develop monitoring and evaluation frameworks on our Knowhow pages. If you would like some support with developing or using a theory of change, NCVO Charities Evaluation Services can help.

Contributor

Page last edited Apr 04, 2019

Help us to improve this page – give us feedback.

1 star 2 stars 3 stars 4 stars 5 stars 3/5 from 317 ratings