Evaluation to support the social impact journey

CSI associate Rachael Trotman’s applied research and evaluative work supports organisations in their social impact journeys. Her goal is to help them ‘know and show’ how they are effecting positive change in the world. Rachael helped found ANZEA – New Zealand’s professional evaluation association in 2006, and continues to contribute to the development of the profession in Aotearoa. This blog is based on a presentation she made to Social Enterprise Auckland, a group of social enterprises and their supporters working for sustainable social change in the region.

Evaluation helps us to convey the realities and complexities of social change work and highlights values and actions underpinning that work. It aims to show what was achieved and what was not, for whom and to what extent. It provides insight and learning to inform practice and action. The best evaluation also gives a sense of the values or principles that made the difference and the people dynamics that helped or hindered.

What is at the heart of evaluation? Here is a key adapted from a Kate McKegg presentation which help us get to grips with it:

E – Valu – Ation

E is for evidence – the data, the information about what you do and what it produces. This includes qualitative and quantitative evidence.

V is for value - the judgement you make on the merit or significance of your activity.

A is for Action – there should be clear implications for action and decision making from any evaluation process.

Why do we need to evaluate? We do it to see if what we are doing is making a difference, to whom, and to what extent. It helps us check that we are working in a way that is consistent with our values and purpose. It gives us information about what is working (what to continue to do), what could be improved (what we can adapt), or what we need to let go of or change. That in turn builds confidence and credibility, which helps people to access resources, enables us to be more strategic and influential, and to collaborate and form partnerships.

Here are two Do-It-Yourself frameworks for evaluation. The first uses this basic formula: what, so what, now what? This is the basic formula for most evaluation.

The “what?” is your project plan, business case, or funding application. Here, you identify what you plan to do, the difference you hope to make, and how you are going to gather evidence to demonstrate that difference. This is where you plan your evaluation.

“So what?” is the evaluative part. This is where you implement your data gathering plan and analyse that data. Ideally the analysis occurs through a group process, rather than one person making sense of it alone. The quality of your evidence is the key to your ability to make claims about impact.

“Now what?” often gets neglected. Applying your findings and learning to do better in the future is important because it informs how you communicate and influence.

The second DIY framework is Success, Evidence, Strategies. Te Awa Ora Trust in Manurewa first alerted me to this strengths-focused framework, which was developed by Kataraina Pipi. It aims to draw out evidence of the values, strategies, and practices that are driving success.

Success is about asking what was successful and for whom, what was not and why?

Evidence is about demonstrating how we know what we did was successful.

Strategies is about clarifying the actions, strategies, and principles that made a difference.

Learning about evaluation can be found online, through events and workshops or involvement with professional associations such as the Aotearoa New Zealand Evaluation Association, and Pasifika Fono for Pacific evaluation. Further evaluation resources are given below.

Evaluation resources

What Works was developed in 2015/16 by myself, Manu Caddie and Community Research. Aotearoa-focused, the website includes links to kaupapa Maori evaluation, local evaluation examples, tools, templates, guidance on ethics, and global resources. 

Inspiring Impact is a UK website aiming to support charities and social enterprises to develop good impact practice. It includes a self-assessment tool which asks ten questions to help you work out what data you need to collect for any specific project. It also has a jargon buster list and a cycle of good impact practice based around plan, do assess and review. 

Better Evaluation is an international collaboration international collaboration to improve evaluation practice and theory by sharing and generating information about options (methods or processes) and approaches. 

Tamarack Institute is Canadian, focused on supporting changemakers to achieve greater impact. 

FSG is a USA-based consultancy that works with organisations addressing social change. A particularly useful resource on their website is an article, The Water of Systems Change, that provides a helpful rubric or filter to understand whether and how you are influencing systems change. 

This blog was developed around this PowerPoint presentation - Getting to Impact