Closing date: 14 Dec 2015
Terms of Reference:
Oxfam effectiveness in building resilience: Synthesis report for the Resilience Outcome Area
1.PURPOSE
This assignment will focus on conducting an analysis of a set of ex-post impact evaluations (called “Effectiveness Reviews”) and final evaluations of several projects aiming to build resilience in order to capture the effectiveness of interventions supported by Oxfam to generate change in resilience. Results of this assignment will inform on whether Oxfam is contributing to achieve their change goals defined in the Oxfam Strategic Plan 2013 -2019.
2.BACKGROUND
Evaluation of the Oxfam Strategic Plan
The Oxfam Strategic Plan[1] establishes the priorities for Oxfam for six years, 2013-2019. These priorities are set out in six change goals that will guide Oxfam interventions over this time period to reduce poverty and bring greater justice. The six change goals are: Goal 1: Right to be heard: People claiming their right to a better life; Goal 2: Advancing gender justice; Goal 3: Saving lives, now and in the future; Goal 4: Sustainable Food; Goal 5: Fair sharing of natural resources and Goal 6: Financing for development and universal essential services.
In order to assess whether Oxfam is contributing to these goals, Oxfam Senior Leadership identified indicative outcomes areas under each change goal that will be used to demonstrate Oxfam’s effectiveness in promoting positive change. An additional outcome area was defined to capture Oxfam effectiveness in building resilience, acknowledging that resilience is crosscutting issue particularly relevant for Change Goals 3, 4 and 5.
The indicative outcome areas identified are:
External Change Goal
Indicative outcome area
1.Right to be Heard
Transforming power relations
2.Gender Justice
Gender Based Violence
3.Saving Lives
Quality of response judged against sector standards
4.Food
Income and food security
5.Natural Resources
Secure access to & control of natural resources
6.Financing for Development
Financial flows to essential services
7.Cross cutting
(change goals 3, 4, and 5)
Resilience
Oxfam defines effectiveness as a positive contribution to change, and will not seek to assess the proportion of observed change which can be attributed solely to the evaluated interventions. Though these indicative outcome areas are not intended to represent the totality of the work being done by Oxfam (under each change goal or outside) they focus on meaningful areas where Oxfam will be able to present a credible summary of its effectiveness promoting positive change.
The Oxfam effectiveness on each Indicative Outcome Area will be measured twice during the OSP lifespan. The first report will cover the period from January 2013 to October 2015 (this corresponds to the present assignment). A second report will be done by 2019 and will cover a period from November 2015 to December 2018.
Resilience in Oxfam
‘Resilience’, which Oxfam defines as ‘*the ability of women and men to realise their rights and improve their well-being despite shocks, stresses, and uncertainty*’[2], has rapidly become one of the most important concepts for development work (Schipper & Langston, 2015). Although there is still some variation across development practitioners over how to define resilience, the concept is gradually becoming more and more consistent throughout the sector (Béné, Frankenberger, & Nelson, 2015).
These challenges are especially acute for Oxfam, because at this time there are different practices and ways of framing resilience being deployed across the organisation. For example, the resilience approach developed by Oxfam Great Britain (OGB) to measure resilience in the deep dive impact evaluations known as ‘Effectiveness Reviews’, resilience has been conceptualised in terms of five dimensions which can be used to generate a set of household-level indicators[3].
Figure 1. Five dimensions of resilience used in the Effectiveness Review
However, Oxfam has recently developed a set of guidelines for designing and implementing programmes, in which resilience is conceptualised differently. Rather than understanding resilience in terms of five dimensions, instead it is framed in terms of three capacities – absorptive, adaptive, and transformative. This approach is summarised in Box 1, and is described in more detail in the programme policy guidelines prepared by Jeans, Thomas, and Castillo, attached in Annex A. It appears, overall, that using the three capacities to guide programme design and implementation brings Oxfam in line with other actors in the development sector.
Box 1: Absorptive, Adaptive, and Transformative Capacity
Absorptive capacity– the capacity to take protective action (including preparedness) and to cope with shocks and stress. It is needed as shocks and stress will continue to happen, for example due to climate variability, protracted conflict and extreme weather events.
Adaptive capacity– the capacity to make adjustments and incremental changes in anticipation of or in response to change, in ways that create more flexibility in the future. It is needed as change is ongoing and uncertain, and because systemic change takes time and sustained engagement.
Transformative capacity– the capacity to change systems that create risk, vulnerability and inequality. It is needed to address the drivers of risk, vulnerability and inequality and because social and natural systems are themselves being transformed, for example by globalization and climate change.
The meta-analysis of Oxfam Effectiveness Reviews
Parallel to the present assignment Oxfam will conduct a quantitative meta-analysis of the ex-post impact evaluations of Oxfam’s resilience-building projects (Effectiveness Reviews). Based on the quantitative analysis of data sets gathered through household level surveys, this meta-analysis, will help Oxfam understand (1) the aspects of resilience that are most affected by its projects, and (2) how well current (and future) measurement approaches capture particular resilience characteristics. Indeed, some of the key questions that the meta-analysis will try to address are closely related with the ones that this assignment will also analyse.
Please see the full ToR of the Meta-analysis of Oxfam Effectiveness Reviews in Annex B.
3.OBJECTIVES OF THE STUDY
The main purpose of this exercise is to capture the effectiveness of interventions supported by Oxfam (either directly or with/through partners) to generate change in the indicative outcome area of resilience.
Thus, the specific objectives for this study are:
‒ Synthesise what selected project evaluations tell us about Oxfam effectiveness in building resilience based in Oxfam’s new framing of resilience
‒ To identify which are the good practices, limitations and/or gaps of selected evaluated projects in building resilience, and provide recommendations on how to improve Oxfam’s resilience interventions to maximize effectiveness.
4.KEY RESEARCH QUESTIONS
The Senior Leadership of Oxfam established in 2013 the main question that should be addressed to inform the Resilience Outcome Area of the OSP. This question was: Have Oxfam’s interventions contributed to households demonstrating a greater ability to minimize risk from shocks and adapt to emerging trends and uncertainty? If so, why? If not, why not?[4]
Considering the new Oxfam Framework for Resilient Development that conceptualise resilience as the enhancement of absorptive, adaptive and transformative capacities all together and a different levels, this initial question has been adapted to ensure alignment with Oxfam current understanding of resilience. Thus, the main question this assignment will address is:
1.To what extend is there evidence that Oxfam’s interventions contributed to build absorptive, adaptive and transformative capacities together at multiple levels (individuals -women, men-, households, communities, municipalities or wider systems (regional or national)?If so, why? If not, why not?
In order to explain the why or why not studied interventions have built resilience capacities these secondary research questions should be addressed:
2.To what extent were Oxfam’s interventions informed by a multi-dimensional integrated analysis of risks, fragility, vulnerability and capacities?
3.To what extent did Oxfam’s interventions use different building blocks (that is established areas of practice such as CCA, NRM, DRR, livelihoods, market development, WaSH, etc) to build absorptive, adaptive and transformative capacity? If so, how were they integrated and sequenced?
4.On which social change processes and multi-stakeholder collaborations were Oxfam’s interventions based on building resilience capacities?
Some of the key research question of the Metaanalysis of Oxfam Effectiveness Review are contributing to the above questions, specifically these are:
How does the existing framework for measuring resilience in the effectiveness reviews (five dimensions) map onto Oxfam’s new approach for programming (three capacities), and what might be missing?
What overall impact has Oxfam had on resilience? Are there any systematic differences between Oxfam’s impacts in certain contexts?
What types of indicators are driving changes in the overall resilience index and is it possible to identify a few key levers of change? Are these levers of change focussed within particular dimensions or within particular capacities? (E.g. have improvements in the resilience index resulted only because Oxfam projects have increased components of absorptive capacity?)
5.METHODOLOGICAL CONSIDERATIONS
The study will consist of a review and critical analysis of 15 ex-post impact evaluation of resilience building projects and 10 other final evaluation reports of selected projects aiming to build resilience. In doing this, the study will carry out a sense making exercise and synthesis of the evidence of change (outcomes) assessed by these evaluations. Please see in Annex C the list of evaluation reports that will be analysed where information is facilitated concerning countries/regions and methodologies used)
This analysis should inform about the levels of evidence found (acknowledging that different evaluation methodologies used may result in different levels of evidence) and use the Oxfam’s resilience analytical framework[5].
The evaluation reports to be reviewed can be divided into two categories according to the methodologies that were applied:
Category 1: 15 evaluation reports taken from Oxfam GB’s series of Effectiveness Reviews (http://policy-practice.oxfam.org.uk/our-approach/monitoring-evaluation/effectiveness-reviews). These are ex-post impact evaluations, carried out on a random sample of large Oxfam projects, after they have reached maturity.[6] The analysis uses quasi-experimental methods to assess Oxfam’s effectiveness in building resilience. All 10 evaluations use the same methodological approach that conceptualises resilience in terms of five dimensions to generate a set of household-level indicators[7].
The core source of information to consider for the analysis of Effectiveness Review will be the ER reports and the raw household survey datasets. It is expected that the quantitative analysis of ER datasets will be done in coordination and close collaboration with the consultant in charge of the Meta-analysis of Oxfam Effectiveness Review (see ToR in annex B)
Category 2:** 11 evaluation reports are final project or programme evaluations that use a set of methodologies to evaluate projects according to commonly used evaluation criteria (pertinence, effectiveness, efficiency, sustainability, etc) and define multiple evaluations questions that respond to specific evaluation objectives defined by programme teams. The data collection methods used in these evaluations includes mainly household surveys, focus group discussions, key informant interviews and participatory approaches and methods. The level of confidence of evidence gathered from these evaluations may vary depending on chosen methodology and sample.
The core source of information to consider for the analysis will be the evaluation reports themselves.
Taking into consideration the above, the consultant will design their own methodology in order to carry out this piece of work. However, we expect that the proposed methodology will include the following activities:
‒ First review of evaluation reports - other than effectiveness reviews - and critical analysis of its quality (rigour, sample size, level of evidence) in order to assess if all the reports have the sufficient quality to be included in the analysis. To judge that evaluations are methodologically sound the following criteria should be considered:
ü The evaluation questions (and sub-questions) logically follow from the evaluation objectives
ü The methodology is explained clearly; its limitations for addressing the key questions are clearly identified
ü The report disaggregates the experiences and feedback of women and men, girls and boys, and marginalized groups
ü Conclusions are supported by evidence and references are provided
ü The recommendations follow logically from the conclusions
ü Recommendations/conclusions target gender justice issues, including power relations
The consultant will do the analysis and provide results and recommendations to Oxfam, but Oxfam will take the final decision on what evaluation reports will be considered for the synthesis and sense making report.
‒ Intermediate feedback to Oxfam on results of quality assessment of evaluation reports, and final decision on sample of evaluation reports to be included in the analysis.
‒ Deep review and analysis of Effectiveness Reviews reports and household level data sets, treatment and preliminary analysis. The review and analysis of ER reports and datasets will be done in close coordination and collaboration with the consultant in charge of the Meta-analysis of Oxfam Effectiveness Review that will be being conducted at the same time, in order to ensure alignment and complementarily (taking advantage of analysis done in the Meta-analysis and avoiding duplication). See in Annex B the ToR for Meta analysis assignment.
‒ Deep review of other final evaluation reports, treatment, and preliminary analysis.
‒ Intermediate feedback to Oxfam on preliminary findings and existing gaps. Consideration on whether it is needed to review other sources of information (e.g. other project key documentation, key informants interviews)
‒ Collection and review of complementary available documentation and/or key informants interviews, treatment and analysis
It is critical that proposed methodology ensure consistency and rigour in the way information is treated and analysed.
We are open to consider tenders for both assignments (Resilience Outcome Area OSP and Meta-analysis of Oxfam Effectiveness Review). In this case the proposed methodology should consider methodological consideration for both researches. (See ToR of Meta-analysis of ER in Annex B, section Methodology)
6.EXPECTED OUTPUTS AND DELIVRABLES
The main deliverable will be a report written in clear, accessible language that will cover the following elements:
ü 2 page executive summary that should provide a brief overview of the exercise (including some information about the types of programs that were reviewed and in which countries), and present significant findings, lessons, conclusions, and recommendations.
ü Background and brief description of the Resilience Outcome Area
ü Purpose of the study: include the objectives and key research questions.
ü Methodology used to generate the report including methodological challenges and limitations that have to be considered to understand and interpret findings.
ü Key findings, including:
o Evidence on final outcomes / changes in building resilience of studied projects, including extend and nature of those changes, relating them with absorptive, adaptive and transformative capacities. Including intended and unintended outcomes (positive or negative)
o Information on quality of reported evidence on outcomes, according to applied methodologies.
o Reasons for these changes and/or for not having evidence of changes
o Information on how studied projects have framed resilience and worked to build resilience (see secondary research questions)
ü Conclusions, including information on good practices, limitations and/or gaps of studied projects in building resilience and demonstrate effectiveness
ü Recommendations, including recommendations for Oxfam how to improve resilience programming and how better demonstrate effectiveness in building resilience
ü Bibliography / References
Othe*r* aspects to consider when writing the report are:
‒ The report should be written in Arial 11.
‒ The title page must include the title of the report, the name of the author(s), the name of the consultancy firm (if that is separate), and the date
‒ Acknowledgements may be noted on a separate page
‒ Include a full list of acronyms and abbreviations
‒ The footnotes should be cited at the bottom of the pages
‒ The bibliography must include references to all documentation considered in the report and should be included at the end of the report
‒ Each bibliographic citation must be structured in the following way: Author’s last name, author’s first name (year) Report Title in Italics. Oxfam affiliate, month of report.
7.TIME COMMITMENT
The assignment is expected to take approximately 25-30 working days to complete. The work should be finalized by February 5th 2016.
This deadline should be respected as Oxfam has de commitment to submit Outcome Area’s reports to the Oxfam Leadership by February.
8.WORK PLAN
The proposal must include a detailed work plan, explaining with phases and deadlines for certain results (see methodological considerations).
If the tender refers only to this assignment, the work plan should detail the key moments where collaboration and coordination with Meta-analysis consultants will be done.
9.BUDGET
The consultant should provide a budget for their work, which breaks down daily fees and any other costs. Budget proposals must not exceed $25.000 or equivalent in GBP (£16,390) or EUR (22,872 €).
10.PAYMENT
Expenses can be paid in advance, but all other consultancy fees will be paid after submission and review of the final
11.LINE MANAGER
Elsa Febles, MEAL Advisor (Email: efebles@oxfamintermon.org)
12.ESSENTIAL SKILLS AND COMPETENCIES
Required skills and competencies for this assignment:
· Relevant university degree and postgraduate / master degree
· Knowledge of resilience thinking and concepts
· Knowledge and experience in resilience programmes or related areas of practice (CCA, DRR, livelihoods, etc)
· Previous experience of conducting quantitative analysis and advanced quantitative and econometric data analysis skills, with expertise in statistical software (Stata preferred, but candidates using MATLAB or R will also be considered)
· Experience and skills on synthesising and conceptually organising information, including quantitative and qualitative data
· Demonstrable experience of conducting large literature reviews.
· Good analytical skills.
· Excellent writing and communication skills
· Fluency read, written and spoken English. Good level of read Spanish.
If tender includes this assignment and also the Meta-analysis of Oxfam ER, please refer also to the Skills and Qualification section in Meta-analysis ToR (see Annex B)
13.SUBMISSION OF PROPOSAL
Consultants wishing to apply should send their proposals via email to Elsa Febles (efebles@oxfamintermon.org) with copy to Jonathan Lain (jlain1@oxfam.org.uk). The proposals must include:
Updated CV of the consultant/s participating in the research
A research proposal that includes:
o A page-cover letter expressing your interest
o The consultant’s research background and significance for this assignment
o A clear methodology
o A detailed work plan, with a clear timeline for completing the document. The work plan should include a schedule for providing intermediate feedback to Oxfam staff, in order to present progress.
o The research proposal should not exceeding 10 pages
- Financial proposal, including a budget detailing daily fees for the consultant/s
We will consider tenders that include both assignments (the present and the Meta-analysis of Oxfam ER), in this case please take into account both ToR (see ToR of Meta-analysis in Annex B) to prepare your proposals.
The deadline for submitting applications is December 14th 2015.
14.TENDER PROCESS
All proposals will be evaluated against the following criteria:
Understanding and adherence to the ToR
Degree of Consultant’s demonstrated skills and competencies according to ToR
Feasibility, coherence and appropriateness of the methodology
Cost-efficiency of the budget, given the available funds
All applicants will be informed about whether their proposals have been selected or not by December 23st 2015.
15.ANNEXES
Annex A. Oxfam Framework and Guidance for Resilient Development
Annex B. Terms of Reference of Meta-analysis of Oxfam Effectiveness Review
Annex C. List of evaluation reports included in the analysis
For further information, please contact: Elsa Febles, MEAL Advisor, Oxfam Intermón: efebles@oxfamintermon.org
[1] Oxfam International (2013), The Power of People Against Poverty: the Oxfam Strategic Plan 2013-19. Available at: http://www.oxfam.org.uk/what-we-do/about-us/how-we-work/our-goals-and-values
[2] Oxfam International (2013) No Accident: Resilience and the Inequality of Risk. Available at: https://www.oxfam.org/en/research/no-accident-resilience-and-inequality-risk (last accessed May 2015).
[3] Hughes K. and Bushell H. (2013). A multidimensional approach to Measuring Resilience. Available at: http://policy-practice.oxfam.org.uk/publications/a-multidimensional-approach-to-measuring-resilience-302641
[4] Oxfam International (2013) Evaluation of the OSP: Outputs and Outcome Areas
[5] Oxfam International (2015) Oxfam Framework and Guidance for Resilient Development
[6] In practice, this means sampling from projects with a financial value of at least 250,000EUR, which have been running for at least 2½ years. The project may either have finished or still be running when Effectiveness Reviews are undertaken.
[7] Hughes K. and Bushell H. (2013). A multidimensional approach to Measuring Resilience. Available at: http://policy-practice.oxfam.org.uk/publications/a-multidimensional-approach-to-measuring-resilience-302641
How to apply:
1.SUBMISION OF PROPOSAL
Consultants wishing to apply should send their proposals via email to Elsa Febles (efebles@oxfamintermon.org) with copy to Jonathan Lain (jlain1@oxfam.org.uk). The proposals must include:
Updated CV of the consultant/s participating in the research
A research proposal that includes:
o A page-cover letter expressing your interest
o The consultant’s research background and significance for this assignment
o A clear methodology
o A detailed work plan, with a clear timeline for completing the document. The work plan should include a schedule for providing intermediate feedback to Oxfam staff, in order to present progress.
o The research proposal should not exceeding 10 pages
- Financial proposal, including a budget detailing daily fees for the consultant/s
We will consider tenders that include both assignments (the present and the Meta-analysis of Oxfam ER), in this case please take into account both ToR (see ToR of Meta-analysis in Annex B) to prepare your proposals.
The deadline for submitting applications is December 14th 2015.
Please contact Elsa Febles (efebles@oxfamintermon.org) to obtain ToR and related annexes