How We Work
Learn how we measure impact and change
Impact measurement for change
Impact measurement at Oxfam Novib is integrated into our programmes to give a voice to citizens. We work on a variety of subjects, research questions, partnerships, stakeholders and methodologies. We measure accountability and attribution, but increasingly place the emphasis on learning, facilitating adaptive management by reflecting on new insights from the impact level at different moments in the project.
The team is comprised of highly qualified researchers that work with campaign-, programme- and project teams inside and outside Oxfam Novib to research the impact of projects in themes relevant to the Oxfam Novib programmatic ambitions. Through the involvement of clients and local partners in the research at all stages, the findings from impact measurement can be quickly followed up and lessons learned can be gathered and implemented in future programmes.
How do we measure impact?
Oxfam Novib uses a balanced approach of qualitative and quantitative methods with a focus on changes in people’s lives. We make these changes visible in an empowering, inclusive, transparent and rigorous way. Which methodology is proposed to use for a study depends largely depends on the exact content of overarching research questions. One way of initiating this is organising a strategic design workshop to formulate the focus of the analysis and define concepts and indicators (based on a Theory of Change).
This can then be operationalised into quantitative or qualitative methodologies and preferably in a combination of both, the so-called Mixed-Methods Approach. After data collection and analysis of data, the team reflects on the analysis and draws out lesson that can inform design of other campaigns, programmes and projects. We analyse the empirical data relying on a solid theoretical framework and ensuring statistical robustness.
Quantitative and qualitative research methods
In our quantitative research we work with a survey questionnaire with validated questions complemented with tailored questions for each specific intervention.
Case Example: Tax in the Occupied Palestinian Territories (baseline)
An example is the baseline survey that we conducted in Occupied Palestinian Territories. In 2016, at the start of a new project that aims to contribute to a fair tax system and ensure citizen participation in setting budget priorities, we asked 1001 people in the Occupied Palestinian Territories, what they know and think about the Palestinian fiscal system and fairness of the system. The subsequent programme and campaign activities use the findings to guide their messages and strategies. In 2019 an end line will be conducted to measure whether changes occurred and whether these can be attributed to Oxfam Novib support.
Comparing change over time in both the target group and the comparision group allows us to identify the impact of the intervention. See graphic below.We work around questions of ‘What changed for the citizens?’. Data is collected from a random selection of people from the target group and a comparison group. We train local interviewers for this purpose, who usually work with a smartphone app. Data cleaning and statistical analysis is done by a team of researchers, resulting in impact reports, learning notes and infographics. The quantitative analysis forms the basis for a reflection with project staff on the results to facilitate learning, stimulate programme improvements and define areas for further research.
Quantitative surveys give a clear picture of areas of results in terms of where you have scored highly, moderately and low. It also brings out areas where more in-depth study has to be done.
-- Kotura James, CEFORD West Nile, report on “Learning from the IMK survey results of Uganda–partners’ reflection” during Survey Reflection Workshop in August 2014
Our qualitative research includes research tools that are focused on gaining a more in-depth understanding of the change process that is described in the Theory of Change, but also on alternative explanations for change. Therefore, we work around questions of ‘Why and how do changes happen?’. Data will often be collected from purposely sampled individuals that represent the target group or certain stakeholders of interest. Data collection is done by well-trained citizens, partner staff or professional enumerators. Analysis is done by a team of researchers, resulting in impact reports, learning notes and infographics. In a reflection workshop with local partners, experts and / or citizens’ findings are interpreted to ensure contextualisation and learning which leads to programme improvements and strategic steer.
Our ultimate aim with impact measurement is to understand the attribution of our programmes on the lives of the people we serve. Although we consider both the quantitative and qualitative methods as valuable methods on their own, we prefer a mixed-methods approach to measure impact. The integration of findings of both methods capitalises on the respective strengths of each approach, and shows the whole picture of why, how, and what change happens. That way we can make sense of how projects make a contribution to a just world without poverty.
What I learnt was that we can achieve long lasting impact if in terms of changing people’s lives, attitudes and practices if we strengthen our advocacy because for the observed changes, it clearly came out that the CSOs advocated for that change.
-- FAWE Uganda, report on “Learning from the IMK survey results of Uganda–partners’ reflection” during Survey Reflection Workshop in August 2014
Our Partners
The Impact Measurement and Knowledge Team adopts a human-centered approach and places citizens at the centre of each endeavor. We work successfully with local partners, who often implement part of the study themselves and take on roles such as enumerators, coaches, coordinators or analysts depending on the research design, the capacities and ambitions of the partners.
The Impact Measurement and Knowledge Team also works together with regional, national and local knowledge institutes and universities (e.g. Oxford University and Wageningen Economic Center for Research). Annually we organise an ‘expert meeting’ where we invite peers from a variety of backgrounds (e.g. development sector, public and private sector) to discuss one or more aspects of measuring impact. Former topics of the expert meetings include the use of evaluation (2016), measuring influencing programmes (2017) and the implication of the data revolution (2018). For each of our Projects, we identify how we can work together and optimise use of findings and other lessons throughout the process.
