WORLD VISION SOMALIA

END OF PROJECT EVALUATION TERMS OF REFERENCE-GODOBJIRAAN INTEGRATED NUTRITION & LIVELIHOOD PROJECTPROJECT NUMBER: 212547

Acknowledgement

World Vision Somalia would like to acknowledge the support and cooperation extended by the project staff at all levels and in sparing time to contribute towards this evaluation design.

We also like to thank the different stakeholders in Somalia (including the district and government departments, partners and local government officials) for their continued partnership and engagement during the project implementation.

Affirmation

This document determines the information needs and objectives of the Integrated Nutrition & Livelihood Project in Godobjiraan and Eyl districts of Puntland. Except as acknowledged and by references to other authors and publications, the design described herein consists of our own work, undertaken to assess the results of the project, generate recommendations and advance learning, as part of the requirements of World Vision’s Design, Monitoring and Evaluation (DME) Learning System. Primary quantitative and qualitative data collected throughout the evaluation process remains the property of the communities and families described in this document. Therefore, information and data relating to the same must be used only with their consent.

Abbreviations

CDR Crude Death Rate

CMAM Community-Based Management of Acute Malnutrition

CNW Community Nutrition Worker

DME Design Monitoring and Evaluation

FGD Focus Group Discussion

GIS Geographical Information System

IYCF Infant and Young Child Feeding

KII Key Informant Interview

MAM Moderate Acute Malnutrition

MoH Ministry of Health

NGO Non-Governmental Organisation

OTP Outpatient Therapeutic Program

SAM Severe Acute Malnutrition

TSFP Therapeutic Supplementary Feeding Program

PD Positive Deviance**

SC Stabilization Center

U5 Under 5

WV World Vision

1.0 Introduction

1.1 Evaluation Summary

Project name: Godobjiraan Integrated Nutrition & Livelihood Project

Project number: 212547

Project results

Project goal: To contribute to the reduction in morbidities and mortalities associated with malnutrition in Nugaal Region, Puntland State

Outcome 1: Effective and accessible treatment for children with acute malnutrition

Outcome 2: Improved local capacity to manage malnutrition and related illness

Outcome 3: Promotion of community-based behavioural change through PD/Hearth, cooking demonstration, child-friendly spaces and dialogue

Outcome 4: Enhanced food security to prevent malnutrition in the community

Project start date: 1 April 2019

Project end date: 30 September 2020

Target population: 15,220 overall cumulative target

Breakdown:

1 Boys and Girls below 5 years: Target 8640

1. Pregnant and lactating women: Target 6480
2. Community Health care workers: Target 50
3. Health facility workers: Target 50

Project location: Godobjiraan and Eyl Districts, Nugaal Region

Evaluation type: End of Project Evaluation

Evaluation purpose

The purpose of this evaluation is to document and inform the stakeholders (donors, partners and beneficiaries) of the project’s relevance, effectiveness, sustainability, the potential impact in relation to project outcomes and the lessons learned.

Primary Methodologies

The evaluation study will adopt a mix of quantitative and qualitative techniques as summarised below:

Quantitative: Caregiver surveys

Qualitative: Key Informant Interview (KII), Document Reviews, Observation

Evaluation duration: TBA

Anticipated evaluation report release date: TBA

Description of Project Being Evaluated

World Vision Somalia (WVS) has worked with the children of Somalia, their families and communities since 1992 through a variety of emergency and rehabilitative programming to address the emergency needs of the communities while addressing some of the underlying causes of vulnerability in those same communities. The strategic aim of the Somalia program is to develop and implement high-quality projects that address the emergency, rehabilitation and developmental needs in a demand-driven and responsive manner. The overall goal of all World Vision programs is to save lives and contribute towards the economic development of the Somali people while contributing to the child well-being outcomes.

In Puntland, a high level of acute malnutrition tends to persist in several population groups due to underlying/structural causes. The overall, level of acute malnutrition at national level has improved to Serious (5-9.9% Global Acute Malnutrition-GAM) in the current 2018 Gu 2018 (median GAM of 14.0%) from Critical in 2017 Gu (median GAM of 17.4%). There are no improvement in the overall current acute malnutrition prevalence compared to 2017/18 Deyr (median GAM of 13.8%). GAM prevalence is Critical (15-29.9%) in 12 out of 33 populations surveyed. Crude Death Rate (CDR) is Critical (1 to <2/10 000/day) in 3 out of 33 population groups surveyed. Morbidity rates remain high (>20 %) in many parts of the country. Accordingly, between August and December 2018 an estimated 294200 children were estimated to face acute malnutrition, including 55200 likely to be severely malnourished (total acute malnutrition burden) nationally. According to the Integrated Nutrition and Mortality Survey conducted in 2018, the global acute malnutrition (GAM) prevalence was recorded at 17.8 % (13.3 – 23.3 95% C.I.) indicating critical situation in Burtinle and serious in Garowe while the prevalence of severe acute malnutrition (SAM) in the survey was at 3.7 %( 2.2 – 6.2 95% C.I.)

Infant and young child feeding practices directly affect the nutritional status of children under two years of age and, ultimately, impact child survival. Improving infant and young child feeding practices in children 0–23 months of age is therefore critical to improved nutrition, health and development of children. The IYCF assessment conducted during the 2018 SMART Survey for Puntland in Garowe and Burtinle districts[1] showed a significant proportion of children (14.7%) having never been breastfed. Generally, 80.2% of children were ever breastfed, and 27% had early initiation of breastfeeding as a reflection of the breastfeeding culture which is low keeping in mind the importance of early initiation of breastfeeding both to the infant and the mother. On exclusive breastfeeding, 9.3% mothers exclusively breastfed up to 6 months of age according to the end of project evaluation to assess the primary health care project Eyl, Puntland, Somalia. These children miss out on the many benefits of breastmilk, specifically missing out on some of the nutrition’s the child would need in the first six months of life, as well as missing out on protection against common childhood infections such as diarrhoea and pneumonia. Poor and untimely complementary feeding is one of the risk factors to child undernutrition. Only about half (52.2%) of the children 6-8 months were introduced to complementary foods on time. All the children in the sample did not achieve the acceptable dietary diversity based on a 24- hour recall. The WHO recommends that children 6-23 months should receive foods from four or more food groups to achieve acceptable diet diversity. The project works towards scaling up the implementation of infant and young child feeding programs to improve the child care practices such as breastfeeding, complementary feeding and dietary diversity for children 6-23 months.

To address the above challenges, WV Somalia has been implementing the integrated nutrition and livelihoods project to provide prevention, treatment and management of severe acute malnutrition within the target region. WV Somalia closely works with the identified and trained CHWs at the community level in identification, referral and follow-up of severely malnourished children from the community to the health facilities for treatment and management. The project also supports the stabilization centre-crucial in the management of severely malnourished under-fives with complications and the 5 Outpatient therapeutic program (OTP) in management of severely malnourished under-fives without complication. The project provides referral mechanisms from the OTP program to the WFP supported Therapeutic Supplementary Feeding Program (TSFP) program for a continuum of care to recovery. Besides, the project works to strengthen the active case finding for the PLWs, through the Mother led MUAC and the CHWs and ensure that these are entered into the WFP Nutrition that focuses on the TSFP program. Similarly, the project through the Positive Deviance /Hearth (PDHearth) component focuses on identification, referral and management of severely malnourished under-fives who have been followed throughout the project period to assess progress. Community mobilization and sensitization are conducted by the CHWs on the project services to create awareness and enhance uptake. Community dialogue meetings are also held to discuss the key barriers to proper nutrition and suggest new ways to counter the barriers, thus contributing to improved nutrition situation of the target population.

The project has been implemented for the last 18 months and will come to an end on 30 September 2020. Therefore, this ToR has been prepared to hire a highly competent external Consultant to evaluate the project performance in the implementation period.

2.1 Project goal

The main objective of the project is to contribute to addressing the underlying and immediate causes while reducing morbidities and mortalities associated with malnutrition in Nuggal Region, Puntland.

Project Results Framework

Key results: Outcome 1: Effective and accessible treatment for children with acute malnutrition (through the Integrated management of acute malnutrition (IMAM) approach)

Indicators:

· % of cases discharged as cured

· % of cases Discharged as Default

· % of Cases Discharged as Death

· %of Cases Discharged as Relapse

· % of Cases as non-respondent

· Average Length of Stay in the program

· % of Cases registered as referrals from SC to OTPs

· % of Cases registered as referral from OTPs to SFPs

Key results: Output 1.1: Children (6-59 months of age) with severe acute malnutrition (SAM) plus medical complications are treated through stabilization care (SC)

· Number of children (boys and girls) screened for malnutrition from all contact points at the community, Health facility and Mobile clinic level

· Number of children months (Boys and Girls) admitted into stabilization care (SC)

· Number of discharged cases (to OTP) who recovered from SC

· Number of cases who defaulted from SC

· Number of discharged cases due to death from SC

· Number of discharged cases who did not recover from SC

· Average duration of stay at SC

Key results: Output 1.2: Children (6-59 months of age) with severe acute malnutrition (SAM) and no medical complications are treated through OTP

· Number of children aged 6-59 months admitted into OTP

· Number of discharged cases who defaulted from OTP

· Number of discharged cases due to death from OTP

· Number of discharged cases who did not recover from OTP

· Number of discharged cases who recovered from OTP

· Number of discharged cases (referred to SFP) who recovered from OTP

· The average duration of stay at OTP

Key results: Output 1.3: Children (6-59 months of age) with moderate acute malnutrition and no complications are treated through SFP

· Number of moderately malnourished (MAM) children (6-59 months) admitted to SFP

· Number of discharged cases who recovered from SFP

· Number of discharged cases who defaulted from SFP

· Number of discharged cases due to death from SFP

· Number of discharged cases who did not recover from SFP

· Number of moderately malnourished children admitted to SFP as referrals from OTPs

Key results: Output 1.4: Pregnant and lactating women (PLWs) are treated through SFPs

· Number of pregnant and lactating women screened for malnutrition

· Number of PLWs admitted into SFP

· Number of discharged cases (PLWs) who recovered from SFP

· Number of discharged cases (PLWs) who defaulted from SFP

· Number of discharged cases due to death from SFP

· Number of discharged cases who did not recover from SFP

IKey results: Outcome 2: Improved local capacity to manage malnutrition and related illness

· % of service providers identified as having a capacity gap in the IMAM Approaches

· % of Health workers in the Operating Health facilities trained on IMAM approaches (OTP, SC, SFP,)

· % of Health Workers in the Operating Health facilities trained in IMAM approaches offering IMAM related services.

· % of CFWs in the operational Area trained on IMAM Approaches supporting the implementation

· % of Mothers trained in the Mother Led MUAC initiative who refer malnourished cases for management.

· % of Service Providers who show improvement at technical Support supervision visits.

Key results: Output 2.1: Enhanced capacity of health care workers and community Family Volunteers to offer quality services to the target population

· Number of health workers (MoH) and WVS trained or refreshed in IMAM Approaches

· Number of community Family volunteers (CFVs) trained in IMAM approach

· Number of CFWs engaged in community outreach for acute malnutrition

· Number of Health workers mentored through joint supportive supervisions

· Number of Review meetings conducted during the project period.

· Number of Health Facilities participated in a services availability and readiness assessment

· Number of Mothers Trained in screening malnourished Children through the Mother Led MUAC

· Number of CFWs trained on Learning through Play

Key results: Output 2.2: Survival of infants and young children through optimal infant and young child feeding practices

· Number of Mother led Nutrition and Health support Groups sessions conducted in the Villages.

· Number of IYCF Sessions conducted by the CFWs

· Number of IYCF counselling sessions conducted by CFWs

· Number of Mothers of children 0-23 months who have received counselling, support or messages on optimal breastfeeding at least once in the reporting Period

· Number of Children Receiving Vitamin A in the reporting period

· Number of Mothers referring children through the Mother Led MUAC

Key results: Outcome 3: *Promotion of community based behavioral change through PD/Hearth, cooking demonstration, child friendly spaces and dialogue*

· % of mothers with severely malnourished children aged 6 to 36 months completing all the PD/Hearth sessions

· % of children admitted in the Hearth sessions who gain the required weight after completing the Hearth sessions

· % of Children involved in play activities during the Hearth sessions

· % of mothers who can afford to bring food for cooking at least twice during the Health sessions

· %of Mothers from the Hearth sessions referred to a Mother Led Support group

· % of Mothers from Hearth sessions referred and linked to an IGA

Key results: Output 3.1: Sustained community efforts in the rehabilitation of malnourished children through the PD/Hearth initiative

· Number of caregivers with malnourished children attend PD Hearth sessions for 12 days

· Number of children participating in PD/Hearth sessions who gain 400 grams in one month

· Number of enrolled caregivers who prepare PD/Hearth menu at home

· Number of participating children who continue their growth trajectory at six months

· Number of visits a PD hearth household received during the 6 months following the session

· Number of children who participate in play sessions during the Hearth sessions

· Number of Mothers referred to IGAs after the Hearth sessions

· Number of Mothers who are referred to Mother Led Support Groups after Hearth sessions.

Key results: Outcome 4: Enhanced food security to prevent malnutrition in the community

· % of mothers from Hearth sessions enrolled in VSLAs

· % of VSLAs formed from communities own capital resources

· % of VSLAs receiving less than 50% share capital from World Vision as inputs

· % of Mothers from Hearth sessions initiating Kitchen gardens

· % of Mothers with Kitchen gardens with only seed input from World Vision

· % of Villages with a high chance of having IGAs surviving without World Vision input on a feasibility index.

Key results: Output 4.1: Increased availability of food, better nutrition, and food diversity through Promotion of Family gardening and VSLAs within the target population

· Number of hearth mothers/caregivers dialogued on kitchen gardening during the Hearth sessions

· Number of drama sessions conducted during the implementation period focusing on kitchen gardens and income-generating activity (IGA)

· Number of IGA’s established from the VSAL Approach

· Number of VSLA starting to operate on communities own capital

· Number of VSLA getting the minimal set top-up from World Vision

· Number of VSLA getting more than the Minimum top-up from World Vision

· Number of Households with kitchen Gardens

· Number of Mothers who initiates kitchen gardens with World Vision inputs

3. Evaluation Target Audiences

The project evaluation is intended to benefit multiple stakeholders that have been involved directly or indirectly in the project implementation process. In particular, the following stakeholders will be involved in the evaluation process:

• Project beneficiaries
• Community Nutrition Workers (CNWs)
• Ministry of Health (MoH) of Puntland and the various departments
• Local government authorities in Eyl and Godobjiraan districts
• Local and international organizations that are operating within Eyl and Godobjiraan Districts
• Support Office, World Vision Hong Kong
• WVS staff, among others.

4. Evaluation Type

This is an End of Project Evaluation that will assess the progress made by the project towards achieving the project goal of contributing to addressing the underlying and immediate causes while reducing morbidities and mortalities associated with malnutrition in the Nuggal Region. The assessment of the project’s impact will focus on the contribution made by the project from inception.

5. Evaluation Purpose and Objectives

The primary purpose of this evaluation is to assess the impact, appropriateness, effectiveness, efficiency, and sustainability of the project. The project endline evaluation will also help to draw key lessons learned and the best practices to the project stakeholders. In particular, the project evaluation will be shaped by the following specific key evaluation questions:

Objective: Impact

Key Evaluation Questions

· What has been the impact of the project interventions on the community?

· Besides, establish the level of impact on the target beneficiaries.

· What contributions have made the project’s interventions on the impact measured on the community and the target beneficiaries?

· What other factors and actors contributed?

Objective: Relevance

Key Evaluation Questions

· What was the level of community involvement and participation in the project design, including; goal setting, planning, implementation, and monitoring?

· How equitably has the project benefited the; women, men, boys and girls?

· How appropriate was the project design to the needs of the community?

Objective: Effectiveness

Key Evaluation Questions

§ What are the achievements against set objectives/targets?

§ Compare actual with planned outputs and how have outputs been translated into outcomes.

§ The evaluation shall also establish the possible deviation from planned outputs and likely outcomes.

Objective: Efficiency

Key Evaluation Questions

• How adequate were the available resources qualitatively and quantitatively?
• Were all the project resources utilized optimally?
• Explore alternative low-cost approaches that could have been used to achieve similar results?
• How could the efficiency of the project be improved without compromising outputs?
• Assess the timeliness of implementing the project activities.
• How adequate were the reporting and monitoring systems of the project?
• Have the project outputs been achieved at a reasonable cost?

Objective: Sustainability

Key Evaluation Questions

• Are there sustainability plans, structures and skills in place to ensure there is the sustainability of project benefits? How adequate are they?
• How is the community and local partners prepared to continue with the project outcomes?
• How likely are the outcomes to be sustainable and enduring? In what ways will it leave a legacy for its beneficiaries and the communities?
• In what ways are women and men in communities, the local partners and government stakeholder’s partners prepared to continue with the project outcome?

6. Evaluation Methodology

The evaluation methodology will be designed in alignment with COVID-19 Standard Operating Procedures (SOPs) and preventive measures provided by the Ministry of Health (MOH) Puntland. The data collection process will apply both quantitative and qualitative methods. However, a more detailed evaluation methodology will be designed by the external Consultant in consultation with WV Somalia Quality Assurance team and the Project Manager. The detailed design of methodology must include the following;

• The evaluation design (in alignment with COVID-19) preventive measures
• Data collection instruments, protocols, and procedures
• Procedures for analysing quantitative and qualitative data
• Data presentation/dissemination methods
• Report writing and sharing etc.

The key data collection methods will include the following among others.

• Document reviews including the project proposal, monthly, quarterly monitoring reports and project review reports.
• Caregiver surveys
• Remote Key Informant Interviews (KII) with WV Somalia staff and partners
• Limited focus group discussions with strict adherence to COVID-19 SOPs
• Reflection and feedback sessions with staff and partners.

Given the current travel restrictions, the Consultant may be required to remotely work with WV Somalia Quality Assurance team in implementing the evaluation process. Besides, the Consultant will be expected to employ the use of mobile data collection and Geographical Information System (GIS) tools in the evaluation process, ranging from data collection, analysis and presentation of results.

7. Evaluation Deliverables

The Consultant will be expected to deliver the following outputs:

• An inception report detailing the methodology to be used and sample size calculations, a detailed execution plan, data-collection tools.
• Draft report submitted to WVS within an agreed timeline between the WVS and the Consultant (soft-copy)
• A presentation of the key findings and recommendations to WV Somalia and other stakeholders. This will be remotely done, given the current coronavirus pandemic.
• Raw data submitted to WV Somalia alongside the final report.
• Final report (soft copy) submitted to Design Monitoring and Evaluation Manager and the Project Manager. However, the Consultant should note that the final evaluation report shall follow the structure below:

i. Table of Contents

ii. Acknowledgments

iii. Affirmation

iv. Glossary/Acronyms and Abbreviations

v. Executive Summary

1. Introduction/Background
2. Methodology
3. Limitations
4. Findings
5. Conclusions and Recommendations
6. Lessons Learned from the evaluation process
7. Appendices.

8. Lessons Learned

· Orientation and training of both internal and external data collectors, as well as the pilot testing of contextualized evaluation tools, KII and FGD tools, should be given a thorough consideration; hence a considerable time should be planned for these tasks.

· Data collection quality control mechanisms and data validation schemes should be embedded into the evaluation design to ensure that the findings are valid and sufficient for informed decision-making.

9. Time frame

The overall evaluation process is expected to take 30 days, including preparation, data collection, and analysis, and reporting. The Consultant should be able to undertake some of the tasks concurrently to fit within the planned time-frame, without compromising the quality expected.

10. Authority and Responsibility

WV Somalia will establish an evaluation team to oversee all the related tasks. The DME Manager will be responsible for the overall coordination of all the evaluation tasks with the Consultant. Besides, the Project Manager, Regional Operations Manager, and Quality Assurance & Strategy Manager will provide all the necessary technical and operational support required throughout the evaluation process.

Support from WV Somalia

World Vision Somalia will be responsible for the following:

· Provide the key project documents to the Consultant to finalize the evaluation methodology and data collection tools

· Provide input for evaluation study methodology, data collection tools and report.

· Ensure that input from WVS is circulated and shared with external Consultant and vice versa

• The Consultant will be required to lead the evaluation remotely and in close collaboration with WV Quality Assurance team, however; only where necessary, cater for the travel expenses involved to the field location.
• Supervise the data collection process
• Vehicle hire to support the evaluation exercise
• Recruitment and payment of the enumerators and translation of questionnaires, where necessary
• Stationery for data collection**

· Overall accountability of the evaluation process

· Guidance and coordination throughout all the phases of evaluation, keeping communication with external Consultant throughout all phases

· Closely follow up the data collection process, ensuring quality control, daily debriefing, meeting the timelines set for interview completion;

· Inform the evaluation audience for their involvement in the study and help in setting-up virtual interviews with the key stakeholders.

The Consultant will be responsible for the following:

· Review all relevant documents for the evaluation study

· Develop evaluation study design which includes survey methodology and the data collection tools (questionnaire; focus group guides, interview protocol, data entry templates, etc.), as appropriate, including a field manual for training, in consultation with evaluation team, reflect WVS feedback on the methodology

· Designing the data entry template, procedures and systems, and training of entry clerks in the use of the template,

· Develop the fieldwork schedule in consultation with the evaluation team

· Conduct training for the data collectors during the field visits phase, finalize the evaluation schedule

· Remotely supervise the data collection process, give advice and ensure the quality of the data

· Remotely conduct interviews (KII) with key project staff,

· Data analysis and report writing, draft the first report and include WVS feedback and finalize the report for submission

· Provide required data that is completed and labelled in English (variables and values) for both the SPSS and Microsoft file formats.

· Provide daily field briefing to the DME and Project Managers on the progress and any challenges from the field.

11. Limitations

Devastating and unpredictable spread of COVID-19 throughout the world has caused unprecedented global lockdowns and immense challenges for humanitarian operations. With the travel restrictions and public health preventive measures in place, it may be difficult for the Consultant to travel to the field as well as limiting integration of qualitative techniques using focus group discussions. However, WV Somalia has continued to adapt to the changing situation by employing remote and mobile phone-based data collection. In the case of caregiver surveys, very close adherence to the public health guidelines will be observed.

12. Documents

The key documents to be reviewed for the evaluation study are as follows:

· Project document (needs assessment, proposal, logframe)

· Baseline Report

· Knowledge Attitudes and Practices (KAP) Survey Report

· Monthly, quarterly, semi-annual and annual reports

· Training reports

· Success stories

· Patient registers

· Any district level secondary data and other relevant documents and reports.

13. Qualifications of the Consultant

The evaluation exercise will be undertaken by an external Consultant who will work in close collaboration with the Project Manager and Quality Assurance team. Therefore, we are looking for a Consultant/team with the following skills and qualifications;

· The team leader must possess a post-graduate degree in Public Health, Nutrition, Social sciences or related discipline.

· Strong and documented experience in conducting participatory qualitative assessments related to health and nutrition

· Demonstrated experience in leading at least three similar project evaluation studies such as surveys and group interviews,

· At least 10 years’ experience in conducting baseline/evaluations for complex projects such as livelihood, health and nutrition, water and sanitation, and hygiene implemented by non-governmental and private sector actors.

· A solid understanding of remote learning through mobile technology, child health and child care, water and sanitation.

· Demonstrated experience leading teams.

· Demonstrated experience in training local staff in quantitative and qualitative data collection tools including entry template

· Demonstrated experience in designing survey methodology, data collection tools, processing and analysis of data.

· Ability to interact with host government, partners and/or others as requested by WVS;

· Strong organizational, analytical and reporting skills, presentation skills, attention to detail, ability to meet deadlines, and proficiency in qualitative data analysis software/tools.

· Previous experience in a fragile country with tight security context will be preferred.

• Capacity to use mobile data collection and GIS tools for data collection, and analysis of survey results.

· Excellent verbal and written communication in English required;