
TERMS OF REFERENCE _ END OF PROJECT EVALUATION
WORLD VISION SOMALIA
AREA REHABILATATION PROGARAM (ARP)
June 2023
Introduction
1. Project Summary
Project name (s)
Area Rehabilitated Program (ARP) in Odweyne District of Togdheer Region, Somalialnd
Project goal
Increased resilience of vulnerable communities in Odweyne District of Togdheer Region
Project outcomes
Target Beneficiaries
Population of target area: 75,367
(Direct Beneficiaries: 2865; Indirect Beneficiaries: 17,190)
Sectors
WASH, Health & Nutrition, Peace & Protection, and Education
Project locations
Odweyne district, Togdheer Region, Somaliland
Implementing Location
Villages: Ceelsame, Beerato, Xaaxi, Gatiitaley, Galooley, Qaloocato, Baarcad, and Laanmulaaxo
Project Start Date
November 2020
Project End Date
October 2023
Evaluation type
End of Project Evaluation (External)
Evaluation purpose
The purpose of the End of Project Evaluation is to assess the progress made towards the achievement of the project goal, objective(s) and outcomes on the lives of children, families and communities in the project area with reference to the OECD/DAC evaluation criteria, particularly focusing on relevance, effectiveness, efficiency, and impacts in relation to project results. The evaluation will also help to draw key lessons learnt and the best practices to the project stakeholders.
The evaluation should document evidence of impact, and to highlight and strengthen best practices emerging from the project. World Vision would also like the evaluation to assess the appropriateness of the approaches/ project models used in the program, as well as the potential sustainability markers that have been achieved through the project’s implementation.
Methodology
The end line evaluation will adopt mixed methods design including quantitative and qualitative techniques as summarised below:
Quantitative:
Qualitative:
Evaluation Period
August 2023
Available Project documents
Project proposal, baseline reports, M&E plan, ITT, Detailed Implementation Plan (DIP), semi-annual reports
2. Project Context and Description
World Vision (WV) Somalia has worked with the children of Somalia, their families and communities since 1992 through a variety of emergency and rehabilitative programming to address the emergency needs of the communities while addressing some of the underlying causes of vulnerability in those same communities. During the last 25 years, the program has grown to over 15 districts spread over three main operational regions of Puntland, Somaliland and South-Central with a liaison office in Mogadishu.
The strategic aim of the Somalia program is to develop and implement high quality projects that address emergency, rehabilitation and developmental needs in a demand driven and responsive manner. The overall goal of all World Vision programs is to save lives and contribute towards the economic development of the Somali people, while contributing to the child well-being outcomes. The Odweyne ARP project implemented by WV Somalia is meant to strengthen household and community resilience for sustained improvements in the wellbeing of children within their families in the Somalia. This was based on the severe challenges experienced at the household level. The ARP model builds local level partnerships around shared goals for child wellbeing, including children’s needs around health and nutrition, education and life skills, child rights and protection, water and sanitation, food security and family livelihoods, and disaster risk reduction. By intentionally facilitating more effective between local groups, Local Non-Government Organizations (LNGOs), government ministries and International (INGOs) working in the ARP area; WV leverages greater commitment and capacity toward achieving key child wellbeing outcomes.
Odweyne Area Rehabilitation Program (ARP) is implemented in Odweyne district, Daadmadheedh region of Somaliland. The overall objective of the Odweyne ARP is to increase the resilience to disaster of vulnerable communities in the 8 villages (Ceelsame, Beerato, Xaaxi, Galooley, Gatiitaley, Qaloocato, Baarcad and Laan-mulaho villages) in Odweyne district. This objective is expected to be achieved through implementation of climate vulnerability interventions which will seek to address and mitigate against the impacts of climate change in Somaliland through strategies that seek to enhance resilience of the communities in Odweyne and also through a holistic and integrated approach to food and nutrition security. To ensure the well-being and resilience of communities to withstand climatic shocks, resilient (internal and external shocks including cyclical droughts and other natural disasters), vibrant economic sectors with particular focus on agriculture and livestock will be established through training of farmers, and agro-pastoralists on Farmer Managed Natural Resources (FMNR). Communal rangelands conservation and restoration through soil and water conservation activities as well as establishing fodder banks will be established to help farmers with pasture and fodder during the dry season. Improved resilience to shocks will also be implemented through training the community in disaster risk management at village level. Increased income and productive assets for small holder famers and pastoralists will also be carried out through establishment of Savings for Transformation groups, animal disease control and surveillance system to safe guard against loss of livestock.
Odweyne ARP also intends to enhance agricultural productivity and the project will address key knowledge, skill and resource gaps among male and female farmers, emphasising sustainable and climate-smart practices, techniques, and principles of nutrition-sensitive agriculture. By applying the knowledge and skills, farmers will increase the production of nutrient rich crops for consumption and/or sale, whilst simultaneously supporting the regeneration and mitigating further degradation of the natural resource base upon which their livelihoods depend.
1. Geographic Scope and Target Population
The project targets villages are Ceelsame, Beerato, Galooley, Gatiitaley, Laan-mullaaxo ,Xaaxi, and Baarcad Villages under the Odweyne District: the evaluation will take place in all the targeted districts. In addition, this survey will assess progress made from the inception and the overall achievement of project outcomes.
2. Context
This evaluation is to be done in relation to the context taking into consideration the changes that took place during the project implementation period (droughts, flooding, and insecurity). The evaluation team will also analyze on how the changes in context affect project implementation and impact including the effectiveness of adaptation efforts made by the implementing partners, communities and other stakeholders.
3. Methodology
This section provides a detailed description of the end-line evaluation process. It focuses on the evaluation design, data collection and analysis and reporting process.
3.1 Scope of the Evaluation
Evaluation data will be gathered from the areas where the project has been implemented in Odweyne. Since this is the summative evaluation, it will assess the progress of the project towards the project outcomes. OECD-DAC evaluation criteria of relevance, efficiency, and effectiveness should be applied to bring out the learning and performance of the project.
3.2 Evaluation Type
This end line project evaluation is aimed at assessing the impact and effectiveness of the program in the target locations with the communities and the local authorities. WVS also proposes a performance evaluation to ensure that the project met its intended goals of restoring the wellbeing of vulnerable communities in Somalia affected by the aftermath of severe flooding and drought in Somaliland and sectoral outcomes and how it can be strengthened in the future. World Vision Somalia will hire an external consultant through a competitive process to carry out a quality project end evaluation.
3.3 Evaluation Purpose and Objectives
World Vision Somalia will conduct this evaluation of the project to identify the impact of the project and to assess the relevance of the project logic, and effectiveness of the project models to realize the project outcomes. It further aims to assess the effectiveness of the integration model of the ARP project.
World Vision would also like the evaluation to assess the appropriateness of the approaches used in the program, as well as the potential sustainability markers that have been achieved through the project’s implementation. The end-line evaluation will also help to draw key lessons learnt and the best practices to the project stakeholders. In particular, the project evaluation will be shaped by the following specific evaluation objectives;
3.4 Evaluation Questions
Impact
I. What was the impact of the project on the intended beneficiaries? (expected or unexpected, positive or negative)
Effectiveness
I. How effective was the project in achieving its intended outputs and/or outcomes? II. What are the achievements against set objectives and targets?
III. Compare end line situation to baseline situation. Compare actual with planned outputs and how have outputs been translated into outcomes. The evaluation will also establish the possible deviation from planned outputs and likely outcomes. Including the reasons for deviations if any.
Relevance
Efficiency
I. How efficient were the project activities? Were all the project resources utilized optimally? II. How could we have done things more cheaply or in a timelier manner?
III. How could the efficiency of the project be improved without compromising outputs? IV. How adequate were the reporting and monitoring systems of the project?
Sustainability
3.5 Evaluation Audience
The end line evaluation is intended to benefit multiple stakeholders that have been involved directly or in directly in the project implementation process. In particular, the following are the key stakeholders that will be involved in the evaluation process;
Environment Research and Disaster Preparedness Authority (NERAD)
3.6 Evaluation Methodology
The evaluation methodology will be designed in alignment with World Vision’s Learning through Evaluation with Accountability and Planning (LEAP) guidelines and principles to ensure the quality of evidence. The data collection process will apply both quantitative and qualitative methods. A detailed evaluation methodology will be designed by the external Consultant in consultation with WV Design Monitoring and Evaluation Manager and the Project Manager who will validate the sampling strategy and procedures. The detailed design of methodology must include the following;
The key data collection methods will include the following among others:
Documents review:
The key documents to be reviewed for the evaluation are as follows:
Quantitative Data Collection
The quantitative Population-Based Survey will be administered among a probability sample of participants and households in the target areas. The sample size, sampling frame, and data collection tools will be designed so that data can be statistically compared to test for differences. The Consultant will be expected to employ use of mobile data collection and Geographical Information System (GIS) tools in the evaluation process; ranging from data collection, analysis and presentation of results.
Focus Group Discussions and KIIs
Focus Group Discussions (FGDs) and Key Informant Interviews (KIIs) will be carried out with beneficiaries and stakeholders to provide additional insights and possible reasons for the observed outcomes. The focus groups will target key stakeholders including beneficiaries, comprising; FSL targeted community members including small holder farmers and pastoralists, women groups, Pregnant and lactating Women (PLW), CHWs, WASH Committees, Protection Committees, and leaders. For purposes of plural investigation, the exercise will be conducted with a broad range of representation within the community to enable triangulation of findings and incorporate wideranging perspectives.
Key Informant Interviews (KIIs) will be conducted with a wide range of stakeholders including; WV Project staff, Government officials from the Ministry of Agriculture, Ministry of Livestock, Forestry and Rangelands, and MoEWR & Min of Gender.
Reflection and feedback sessions with staff and partners.
The consultant should consider conducting sessions with staff and project partners to reflect on project implementation and discuss feedback.
4. Deliverables
The Consultant will be expected to deliver the following outputs:
5. Evaluation Report Structure
Title and Opening pages (front matter)—should provide the following basic information:
Table of Contents-including boxes, figures, tables, and annexes with page references.
List of acronyms and abbreviations
Executive Summary
A stand-alone section of two to three pages that should:
Introduction
This section will;
Description of the Intervention
This section will provide the basis for report users to understand the logic and assess the merits of the evaluation methodology and understand the applicability of the evaluation results. The description needs to provide sufficient detail for the report user to derive meaning from the evaluation. In particular, the section will;
Evaluation Scope and Objectives
This section of the report will provide an explanation of the evaluation’s scope, primary objectives and main questions.
Evaluation Approach and Methods
This section will describe in detail the selected methodological approaches, methods and analysis; the rationale for their selection; and how, within the constraints of time and money, the approaches and methods employed yielded data that helped answer the evaluation questions and achieved the evaluation purposes. The description will help the report users judge the merits of the methods used in the evaluation and the credibility of the findings, conclusions and recommendations. The description of methodology will include a discussion of each of the following:
Findings and Conclusions
This section will present the evaluation findings based on the analysis and conclusions drawn from the findings. In particular,
Findings: This section will present findings as statements of fact that are based on analysis of the data. The evaluation findings will be structured around the evaluation criteria and questions so that report users can readily make the connection between what was asked and what was found. Variances between planned and actual results will be explained, as well as factors affecting the achievement of intended results. The assumptions or risks in the project design that subsequently affected implementation will also be discussed.
Conclusions: This section will be comprehensive and balanced and highlight the strengths, weaknesses and outcomes of the intervention. The conclusion section will be substantiated by the evidence and logically connected to the evaluation findings. The conclusion will also respond to key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to the decision-making.
Recommendations: The evaluation will seek to provide very practical, feasible recommendations directed to the intended users of the report about what actions to take or decisions to make. The recommendations will be specifically supported by the evidence and linked to the findings and conclusions around key questions addressed by the evaluation. This shall also address sustainability of the initiative and comment on the adequacy of the project exit strategy.
Lessons Learned: The report will include a discussion of lessons learned from the evaluation, that is; new knowledge gained from the particular circumstances (intervention, context outcomes, even about the evaluation methods) that apply to a similar context. Concise lessons based on the specific evidence presented in the report will be presented in the evaluation report.
Report Annexes: The Annex section will include the following to provide the report reader with supplemental background and methodological details that enhance the credibility of the report.
6. Roles and Responsibilities
6.1 Consultant’s Roles and Responsibilities
The consultant shall conduct desk reviews of relevant project documents such as proposals, assessments, project budgets, monitoring and assessment reports, and WV guidelines. In addition to the desk review, the consultant shall prepare and submit an inception report with a detailed analysis plan that sets out the conceptual framework to be used in the evaluation, the key evaluation questions including the methodology to be used, work plans and schedules for both quantitative and qualitative aspects of the assignment for review and feedback and approval by WV. Finally, the consultant will prepare and submit the evaluation report to WV. The consultant shall carry out field visits to selected sites; conduct household surveys, interviews and/or focus group discussions and
Key informant interviews with local partners, key stakeholders, households, and herder group
6.2 World Vision’s Roles and Responsibilities
7. Time frame
The overall evaluation process is expected to take 30 days including preparation, data collection, and analysis and reporting. The Consultant should be able to undertake some of the tasks concurrently to fit within the planned time-frame, without compromising the quality expected. The assignment is expected to commence on 1st Aug 2023, with the final evaluation report expected by 30th Aug 2023.
8. Bids Evaluation Process and Requirements
The Selection of the consultancy firm will be made based on cumulative analysis (i.e., mandatory requirement and technical qualifications as follows:
Evaluation Criteria:
I. Mandatory Evaluation Requirements
NB: Bidders who will fail to provide mandatory requirements will not qualify for next stage (Technical Evaluation)
II. Technical Evaluation
Technical Evaluation Criteria
The following criteria will be used to evaluate the proposals
Other criteria:
Qualifications
The consultant must have proven expertise and experience in social research with a special focus on the Environmental science, Agriculture, Agricultural Economics, Health and Nutrition, Development studies, Baseline, end-of-project evaluations, midterm evaluations, and impact assessments and be able to implement the final project evaluations in Somalia following the required procedures. Proof of these is to be provided by submitting, together with the application:
Requirements in detail:
Proposal Contents
Proposals from Consultants should include the following information (at a minimum):
o Consultant/Company Profile
o Description of the Methodology and Sample Size Determination
o Demonstrated previous experience in similar assignments and qualifications outlined in this ToR (with submission of at least two most recent reports)
o Proposed data management plan (collection, processing and analysis).
o Proposed timeframe detailing activities and a work plan.
III. Financial Evaluation
A financial proposal with a detailed breakdown of costs (which shall include professional fees and operational budget) quoted in USD.
The applicable tax amount must be clearly stipulated and separated from the base costs. O
Payment Terms must be clearly indicated
Clarification of Bidding Document
A prospective bidder making an inquiry relating to the tender document may notify WVS in writing at caroline_njuki@wvs.org. WVS will only respond to requests for clarification received no later than 6/07/2023.
How to Apply
All interested bidders are requested to submit their proposal in English and by email to somo_supplychain@wvi.org on or before 6th July 2023. Proposals should be submitted in to world vision attachments, namely. Mandatory Requirements, Technical Proposal and Financial Proposal (Bidders who will combine both technical and financial proposals shall be disqualified)
EMAIL TITLE SHOULD BE; – END OF PROJECT EVALUATION_ ARP PROJECT
Bids received after deadline shall not be considered.