Evaluation Terms of Reference: SHARPEN Final Evaluation
Catholic Relief Services, Somalia
May 29, 2020
TOR Version Number 1.0

I. INTRODUCTION
I.A. Evaluation Overview
Catholic Relief Services Somalia is seeking to engage an external consultant to conduct a final evaluation for the SHARPEN program. The aim of the evaluation is to determine the overall merit and value of the response through using meta-questions to assess the project’s relevance and effectiveness as well as to provide evidence-based recommendations that will inform future programming.

Non-experimental evaluation design will be used to measure the changes brought about by the project intervention to the target beneficiaries in Mogadishu, Afgooye Corridor, Baidoa, Cadaado, Luuq, and Dollow. Probability sampling will be used for the household quantitative survey to ensure the subjects of the population get an equal opportunity to be selected as respondents. A stratified two stage cluster sampling approach will be used to select study participants. For the qualitative data collection, a purposive sampling method will be used to select study respondents, based on the role they played in the emergency response.
Quantitative data collection will sample beneficiary households in project target locations in Mogadishu, Afgooye Corridor, Baidoa, Cadaado, Luuq, and Dollow, that have benefited from the project interventions and will be designed to collect data from household heads and/or their spouses on demographic, socio-economic characteristics of the households and the performance of the selected indicators. The data will be collected using a semi-structured household questionnaire and rely on a mobile data collection application via a remote modality.
Qualitative data collection will target the key stakeholders who helped contribute to project design and implementation. This information will be used to supplement and complement the quantitative data collected from the household interviews to provide an in-depth knowledge on how the intervention has been able to support conflict and drought-affected IDPs in the project target locations. The targeted respondents will include the health centers staff, village committee members, representatives of the community and the other project staff involved in the implementation of the response. The qualitative data collection will be done through Key Informant Interviews (KIIs).
The findings of the report would be used by CRS and its implementing partners to determine achievement against performance indicators and to draw lessons to inform future intervention designs and similar programming.
I.B. Modifications from Proposed Monitoring and Evaluation Plan
Due to the COVID-19 outbreak, modifications were made from the Monitoring and Evaluation Plan proposal submitted to OFDA. These modifications are meant to protect the safety and security of community members and staff, abiding by the principle of “Do No Harm,” while maintaining the OFDA standard for monitoring and evaluation. Below is a summary of the changes made:
Table 1.1. Summary of Modifications from the Proposed Monitoring and Evaluation Plan
Modifications Justification
Removed Focused Group Discussions (FGD) as a tool to obtain qualitative data Due to government instructions and to limit the risk of spreading COVID-19, no FGD would be organized, wherein participants are physically present. Furthermore, given the limited access to virtual meeting platforms by participants, it is likewise impossible to perform virtual FGDs.
Use of Phone Surveys for Quantitative Data Collection To abide by the “Do No Harm” principle, no in-person data collection would be done to avoid the risk of spreading COVID-19 to communities and staff. Phone surveys would be the alternative to obtain quantitative data.
Use of Virtual Meeting Platforms or Phone Calls for Key Informant Interviews To abide by the “Do No Harm” principle, face-to-face key informant interviews would be replaced by alternate means such as virtual meeting platforms and phone calls to avoid risk of spreading COVID-19.
The indicator to measure safe water handling is removed. The indicator removed is “Percent of households targeted by the hygiene promotion program who store their drinking water safely in clean containers.” Direct observation would be impossible since no in-person data collection would be conducted to limit the spread of COVID-19. Thus, this indicator, which relies on direct observation, would not be possible to measure.

The shift to remote data collection would include updates on informed consent protocols to accommodate the change in data collection instrument and use of verbal consent.
I.C. Background: CRS and Implementing Partners

Catholic Relief Services (CRS) – An international non-governmental organization supporting relief and development work in over 99 countries around the world. CRS programs assist persons based on need, regardless of creed, ethnicity, or nationality, and works through local church and non-church partners to implement programs. CRS carries out the commitment of the Bishops of the United States to assist the poor and vulnerable overseas. CRS currently addresses food, water, hygiene, health, nutrition, and protection needs of vulnerable Somalis. CRS has been working closely with and channeling resources and support to local organizations inside Somalia since the 1990s. CRS has been implementing activities in Mogadishu since August 2011, in Baidoa since April 2012, and in Gedo region since 2014 with OFDA, FFP, and other private and external donor funding.
SOS Children’s Villages International – An independent, non-governmental international development organization which has been working to meet the needs and protect the interests and rights of children since 1949. SOS in Somalia began in 1983; and a property provided by the government was chosen as the site for the first SOS Children’s Village and its adjoining kindergarten. The SOS School was later converted into an emergency clinic during the war, and the mother and child clinic became part of emergency relief programming. Today it remains the only functioning maternity ward and gynecological care facility in the country. More recently, the SOS Vocational Training Center was established, which offers training courses for nurses and midwives. SOS has worked with CRS since 2011 on emergency programs to provide livelihood recovery, basic health, and nutrition services to vulnerable IDPs and host communities.

Save Somali Women and Children (SSWC) – SSWC was founded in 1992 by a group of Somali female intellectuals from a cross section of the community and has a longstanding history of promoting women’s rights and advocacy. SSWC has worked in the areas of protection, WASH, and livelihoods, and prioritizes supporting grassroots economic projects for women, enhancing their capacity for advocacy on the issue of Female Genital Mutilation (FGM), providing training to Non-Governmental Organizations (NGOs) and Community Based Organizations (CBOs) on women’s rights, and raising awareness on the conditions of women and girls in Somalia in Baidoa, Mogadishu, Afgooye, Caadado, Dsamareeb and Kismayu.

Trocaire – Trocaire is an international non-governmental organization that works with local partners to support communities in over 20 developing countries with a focus on food and resource rights, women’s empowerment and humanitarian response. Trocaire has been operational in Somalia since 1992 and is one of the few organizations that continued to provide life-saving interventions without pause throughout the chaotic decades of civil war. Trocaire employs a unique, community led approach through all its work that has ensured access and safety for its staff and operations.

I.D. Background: Project Goal and Objectives
Catholic Relief Services received a grant valued at $6,000,079 from OFDA/USAID to carry out humanitarian assistance for IDPs and host populations in South-Central Somalia. The project start date was July 23, 2019 and the project will end on July 31, 2020. The project provides an integrated package of life-saving services to highly vulnerable IDPs, host-communities and hard-to-reach populations in Mogadishu, Afgooye Corridor, Baidoa, Cadaado, Luuq and Dollow. CRS works with three implementing partners: Save Somalia Women and Children (SSWC), SOS Children’s Villages Somalia (SOS), and Trocaire Somalia. The project intended to reach a total of 125,000 beneficiaries, including 106,250 (85%) IDPs. The project has four objectives:
Target populations access comprehensive primary care and have reduced morbidity.
Malnutrition in young children (>5) and pregnant/lactating women (PLWs) is decreased.
The risk of GBV against vulnerable groups (including children) is reduced and survivors of GBV access comprehensive services.
Vulnerable populations access clean water and have improved hygiene.

The project goal, sectors, objectives by sector, and activities are presented in the table below.
Table 1.2. Objective and Approach per Sector
Sector and Amount Health
$1,224,936 Nutrition
$1,091,735 Protection
$2,703,646 WASH
$979,762
Objective Target populations access comprehensive primary care and have reduced morbidity. Malnutrition levels in young children (<5) and pregnant/lactating women (PLWs) are decreased. The risk of GBV against vulnerable groups (including children) is reduced and survivors of GBV access comprehensive services. Vulnerable populations access clean water and have improved hygiene.
Approach CRS and its partners support static and mobile health clinics, who use community health workers to extend service provision to hard-to-reach IDP populations who cannot easily access existing facilities. CRS and its partners use the Basic Nutrition Service Package for Somalia protocol (as recommended by the Somalia Nutrition Cluster). CRS uses a combination of strategies to decrease malnutrition, including household level screening, treatment at health facilities, IYCF and mother-to-mother support groups and targeted nutrition messaging. CRS and its partners work within prevention and response to gender-based violence and child protection and psychosocial support services to achieve this objective. GBV survivors access comprehensive services including medical, legal and psycho-social. Furthermore, the project works to raise GBV awareness and mitigation strategies among target communities. Partners also participate in child protection through the provision of child friendly spaces (CFS). CRS and its partners undertake extensive hygiene promotion campaigns, integrated with nutrition messaging at facilities and HH-level through leveraging partner networks of CHWs and CNWs to delivery integrated hygiene messaging. Furthermore, the project supports the construction of latrines, handwashing stations and shallow wells to benefit both IDP and host community populations.

II. PURPOSE OF THE EVALUATION
II.A. Purpose of the Evaluation
The purpose of the evaluation is to assess whether the response achieved the desired outcomes and produced evidence-based recommendations to inform future programming.
The evaluation will seek to:
Determine achievement against performance of select indicators;
Identify to what extent were beneficiaries actively consulted and engaged in the project;
Draw lessons from the project and results achieved to inform future similar programming.
II.B. Key Audiences and Uses
Table 2.1. Summary of Stakeholder Data Needs and Evaluation Role
STAKEHOLDERS STAKEHOLDER EVALUATION DATA NEEDS AND USE STAKEHOLDERS’ ROLE IN THE EVALUATION JUSTIFICATION FOR STAKEHOLDER ROLE
USAID/OFDA To establish the impact of the OFDA project on the target beneficiaries.
Suggestions/recommendations with the potential to shape future programming. Provision of funds to undertake the evaluation.
Review and approval of the evaluation TOR and report. This is the donor for the project.
CRS To establish the impact of OFDA on the target beneficiaries.
Suggestions/recommendations with the potential to shape future programming.
To assess and evaluate the relevance, efficiency, and effectiveness of the response interventions in terms of its implementation approach and strategies. Development of the evaluation TOR.
Recruitment of the consultant.
Reviewing and approval of the evaluation design and tools upon agreement with the consultant.
Provision of relevant project background materials.
Supervision of the consultant.
Review and approval of the final the evaluation report.
Processing payments for the consultant(s) upon receiving of the final Report.
Make travel, accommodation and per diem arrangements for consultant and team.
Ensure smooth flow of consultancy engagement processes including contractual obligations.
Share the final evaluation report with all key stakeholders, including key project staff, partners, donor representative and government. This is the project prime.
Local Implementing Partners: SOS, SSWC and Trocaire To measure the outcome and impact of the intervention.
Suggestions/recommendations with the potential to shape future programming. Assist the consultant in the implementation of the evaluation methodology as appropriate i.e. recruitment of research assistants, participation in sampling, mobilizing sampled communities, scheduling meetings, interviews, etc.
Timely procurement of logistics as agreed with the consultant. These are the project sub-grantees.
District Health Boards Status of health, nutrition and WASH indicators in their constituencies Receive key indicator results through regular consultation channels with Local Implementing Partners DHBs are local oversight mechanisms for health care service provision

III. EVALUATION QUESTIONS / OBJECTIVES
The evaluation will use meta-questions around relevance, effectiveness, and efficiency of project interventions, in assessing whether the project met. These evaluation meta-questions are provided in the table below.
Below is the list of required indicators, baseline benchmarks and end of targets to be measured by the evaluation.
Table 3.1. Performance Indicators
OBJECTIVE STATEMENT INDICATORS BASELINE VALUE TARGET PROPOSED METHOD
Target populations access comprehensive primary care and have reduced morbidity. Number and percentage of community members who can recall target health education message 1. N/A

1. 80%
   Phone Survey
   Malnutrition levels in young children (<5) and pregnant/lactating women (PLWs) are decreased.. Proportion of infants 0-5 months of age who are fed exclusively with breast milk
   Proportion of children 6-23 months of age who receive foods from 4 or more food groups 2. 0%
2. 9.9%
   1. 35%
3. 20% 2. Phone Survey
4. Phone Survey
   The risk of GBV against vulnerable groups (including children) is reduced and survivors of GBV access comprehensive services. Percentage of children whose parents report improvement in sense of safety and well-being of children as a result of accessing a child-friendly space
   1. N/A 4. 75% 4. Phone Survey
      Vulnerable populations access clean water and have improved hygiene. Percent of people targeted by the hygiene promotion program who know at least three (3) of the five (5) critical times to wash hands
      Number of people directly utilizing improved sanitation services provided with OFDA funding
      Average number of users per functioning toilet
      Number of people directly utilizing improved water services provided with OFDA funding
      Average liters/person/day collected from all sources for drinking, cooking, and hygiene
      Estimated safe water supplied per beneficiary in liters/person/day
      Percent of handwashing stations built or rehabilitated in health facilities that are functional
      Percent of households targeted by WASH program that are collecting all water for drinking, cooking, and hygiene from improved water sources
      Percent of households reporting satisfaction with the contents of the WASH NFIs received through direct distribution (i.e. kits) or voucher
      Percent of households reporting satisfaction with the quantity of WASH NFIs received through direct distribution (i.e. kits), vouchers, or cash
      Percent of households reporting satisfaction with the quality of WASH NFIs received through direct distribution (i.e. kits), vouchers, or cash 5. 82%
5. N/A
6. N/A
7. N/A
8. N/A
9. N/A
10. N/A
11. N/A
12. N/A
13. N/A
14. N/A 5. 80%
15. 5,020
16. 20
17. 4,800
18. 15
19. 15
20. 95%
21. 75%
22. 75%
23. 75%
24. 75%
    1. Phone Survey
25. Phone Survey
26. Phone Survey
27. Phone Survey
28. KII and monitoring data from Trocaire staff
29. KII and monitoring data from Trocaire staff
30. KII and monitoring data from Trocaire and SOS staff
31. Phone Survey
32. Phone Survey
33. Phone Survey
34. Phone Survey

All performance indicators must be reported with appropriate disaggregations as specified in USAID/OFDA’s Performance Indicator Reference Sheets.
Table 3.2. Summary of Evaluation Questions
COMPONENTS OF THE PROGRAM WE WOULD LIKE TO LEARN MORE ABOUT QUESTIONS WE HAVE THAT WE WOULD LIKE ANSWERED EXISTING DATA TO HELP ANALYZE THIS QUESTION FURTHER DATA NEEDED WHO SHOULD BE INVOLVED?
Relevance 1. To what degree did the project meet the needs of target beneficiary populations? Project proposal
Project monthly and quarterly reports
Feedback mechanism data Beneficiary and other stakeholder perspectives on the relevance of the project in meeting their needs. CRS, SOS, SSWC, Trocaire, beneficiaries and other relevant stakeholders
Effectiveness 2. To what extent were the sector specific objectives of the project achieved?

1. To what extent were beneficiaries actively consulted and engaged in the project especially in their ability to provide feedback through partner’s accountability mechanisms? Project proposal
   Project monthly and quarterly reports
   Feedback mechanism data, including design documentation of mechanisms
   Previous donor submitted reports
   Partner indicator tracking tables
   CRS partner accountability assessment documents Population level data on required performance indicators (detailed in tabled 4.1 below)
   Feedback from beneficiaries regarding awareness, accessibility and use of accountability mechanisms
   Beneficiary feedback regarding project outputs, outcomes and community consultation around the project implementation
   Analysis of project performance against indicator targets and changes from the baseline CRS, SOS, SSWC, Trocaire, beneficiaries and other relevant stakeholders

IV. EVALUATION METHODOLOGY
IV.A. Evaluation Design and Approach
The final evaluation will employ a non-experimental design for simple pre-post comparison of results using a mixed-methods approach involving both quantitative and qualitative data. Data collection will involve a quantitative beneficiary household survey; document reviews, including routine monitoring data and project reports; beneficiary and stakeholder interviews. The consultant will use a comparative analysis approach to report on project achievements for selected indicator values.

Quantitative HH survey will be conducted through structured questionnaire, with relevant and appropriate questions, that will generate quantitative information that will be captured numerically and can produce summary statistics such as frequency distributions, means, medians, ranges and other measures of variation which describe the beneficiary in an aggregate way. The quantitative HH survey will be complemented by qualitative methods in the form of KIIs. Secondary data from routine project MEAL system and the previous phase final evaluation and needs assessments conducted in the project should also be used to inform the findings, conclusions, and recommendations in this evaluation. The survey will generate end-line data for the outcome indicators of the OFDA project, which could be used as baseline data for follow-on program where appropriate. In this design, the researcher will not control, manipulatethe or alter the predictor variables or project beneficiaries, but will instead rely on interpretation, observation and interactions to concluded, through correlations.

Due to the current COVID-19 outbreak, the evaluation should not rely on any in-person or face-to-face means, relying solely on remote data collection. Quantitative HH survey should be conducted using phone surveys, while KIIs be conducted using virtual meeting platforms or phone calls.

IV.B. Sources of Data and Data Collection Methods
In order to answer the key evaluation questions associated with this assignment, both qualitative and quantitative data will be elicited by the consultant, through primary and secondary sources. First, the evaluation will assess and draw from secondary data, which will be obtained through review of key project documents: quarterly assessments, monthly reports, monitoring data and project proposals. The evaluation will also generate primary data. The consultant will design and conduct a household survey administered remotely to randomly selected beneficiaries in the target locations, to generate primary quantitative data.

In order to generate primary qualitative data, the consultant will conduct key informant interviews, and any other methods considered appropriate, such as case studies and most significant change stories. Key informant interviews will be purposively selected from project beneficiary and project stakeholder populations. The consultant will conduct key informant interview with SOS, SSWC, Trocaire, and CRS Somalia staff, as well as other key project stakeholders. If possible, all qualitative data will be recorded during the informant interview process and translated from Somali to English, as needed. The consultant should include all raw qualitative key informant interview data, as appendices to the Final Evaluation Report, and categorize, summarize, interpret, and highlight key findings and conclusions from all this data in the Final Report itself.
All of the qualitative data collected through the above methods will be used to triangulate, explain, and create context for the trends and data collected through the quantitative methods. In the Final Evaluation Report, the analysis of all data (primary, secondary, quantitative, qualitative) and findings should be organized around and presented in order to answer the three key Evaluation Questions as summarized in Table 3.2.
Both quantitative and qualitative data collection should be done via remote modality. The quantitative household survey should be done through phone surveys, conducting KIIs through virtual meeting platforms or phone calls only.
To abide by the “Do No Harm” principle especially in the context of COVID-19, any in-person data collection is highly discouraged and should be reconsidered, consulting CRS for any plans to perform in-person data collection. This is to limit the risk of spreading COVID-19 to communities and implementing partner staff.
All contact information to be used should comply to the data protection policies of CRS and of the implementing partners.
IV.C. Sampling Strategy

The survey will employ a stratified two-stage cluster sampling as it is the most efficient way to sample the population given that the beneficiaries are in different districts receiving different interventions.

The districts would be the strata to ensure that every district would be proportionately represented in the evaluation. The first stage of sampling would select sample villages/IDP camps from each district determined by Probability Proportional to Size (PPS) sampling. Households, registered as beneficiaries, would then be selected from these villages/IDP camps by simple random sampling and would be the primary units for the survey.

The beneficiary lists from the implementing partners would be the sampling frame. The lists would contain household contact information and intervention(s) received.

The sample size will be calculated using Cochran’s formula,
n_0=(p(1-p) Z^2)/E^2 (1)

n= n_0/(1+(n_0-1)/N) (2)
where,
n0= sample size
n = adjusted sample size for the total number of beneficiaries
N = total number of beneficiaries
p = variance of the population
Z = Z-score corresponding to confidence interval
E = margin of error

In order to have maximum variance and to have at least 95% confidence, p should be set to 0.5 and the Z-score should be equal to 1.96. The evaluation could allow up to a 10% margin of error. A significant buffer should also be added to the sample size to compensate for households not responding or consenting to the survey.

The quantitative survey will employ a remote modality. Due to the current circumstances of COVID-19, the survey will be done remotely using phones, email addresses, and other digital communication available. In order to increase take-up on the survey, CRS suggests the following actions:
Call the respondents via phone, and if not available, via WhatsApp, and other digital means,
Send text messages to the respondents regarding the phone survey at least five minutes before the first call attempt,
Set-up protocols for failed called attempts (e.g. determining maximum number of attempts, time interval between attempts, system of recording for the attempts, etc.)
Have enumerators (if not using IVR) in shifts in order to place calls during off-hours

The quantitative survey would be used for the following indicators:
Number and percentage of community members who can recall target health education message
Proportion of infants 0-5 months of age who are fed exclusively with breast milk
Proportion of children 6-23 months of age who receive foods from 4 or more food groups
Percentage of children whose parents report improvement in sense of safety and well-being of children as a result of accessing a child-friendly space
Percent of people targeted by the hygiene promotion program who know at least three (3) of the five (5) critical times to wash hands
Number of people directly utilizing improved sanitation services provided with OFDA funding
Average number of users per functioning toilet
Number of people directly utilizing improved water services provided with OFDA funding
Percent of households targeted by WASH program that are collecting all water for drinking, cooking, and hygiene from improved water sources
Percent of households reporting satisfaction with the contents of the WASH NFIs received through direct distribution (i.e. kits) or voucher
Percent of households reporting satisfaction with the quantity of WASH NFIs received through direct distribution (i.e. kits), vouchers, or cash
Percent of households reporting satisfaction with the quality of WASH NFIs received through direct distribution (i.e. kits), vouchers, or cash

KIIs and monitoring data from Trocaire staff will be used for the following indicators:
Average liters/person/day collected from all sources for drinking, cooking, and hygiene
Estimated safe water supplied per beneficiary in liters/person/day
Percent of handwashing stations built or rehabilitated in health facilities that are functional

Given the limitation of only using phone surveys because of COVID-19, there is an acknowledgment that the quantitative data could be biased. This potential bias is brought by surveying only those with phone numbers recorded by project MEAL staff. Furthermore, previous experience in the project areas indicate that beneficiaries have low response rates to remote surveys, which could potentially add more bias. Utmost attention must be given to limit these potential biases. Qualitative data from KIIs and document reviews could give more information, which could counteract the potential biases in the quantitative data.

Qualitative data should be collected to gain further insight into each intervention sectors for each district. Qualitative data would include document reviews and KIIs, utilizing only virtual meeting platforms or phone calls to reduce the risk of spreading COVID-19.

For KIIs, there should be at least one interview from a key CRS staff, one interview from a key staff of each of the implementing partners, and one interview from the camp leaders or key persons in each of the village/IDPs surveyed. If possible, KIIs should be recorded during the interview process and translated from Somali to English, as needed.

The consultant shall determine any other KIIs that are feasible within the framework of mitigating the effects of COVID-19 and that would contribute to answering Evaluation Questions 1 and 3 above.

IV.D. Data Analysis Procedures

The study proposes to collect data using household questionnaire coded in mobile application. Descriptive statistics (frequencies and percentages, means, medians, and standard deviations) will be used to describe the evaluation findings on the project indicators.
Qualitative data, obtained using KII will be captured using KII guides, translated into English and later typed in Microsoft Word templates. Analysis for this data will be mainly through content analysis and establishment of themes. The findings from qualitative data will be used to triangulate the primary quantitative data and secondary data (from project documents and performance reports) for the final conclusions of the evaluation findings.
IV.D. Considerations/Recommendations on COVID-19

As COVID-19 affects the communities and stakeholders we work with, CRS is putting the safety and well-being of the staff, beneficiaries, consultants, and other stakeholders in project regions. The consultant is expected to adapt their evaluation plans and methodologies according to the changing situation, prioritizing safety and informing CRS of all changes. The consultant must respect all COVID-19 directives issued by the Federal Government of Somalia and federal member states wherein data collection is taking place.

The consultant is expected to utilize virtual meeting platforms in conducting key informant interviews, especially for the target respondents from CRS, partners, and stakeholders with enough digital literacy. Other respondents who are unable to perform virtual meetings will use phone calls. CRS and implementing partners may assist key stakeholders to participate in telephonic KIIs as needed.

Lastly, the consultant is also expected to be transparent on the possible effects of the COVID-19 situation on the data gathered.

V. EVALUATION TEAM
Evaluation (Lead) Consultant: Will plan and coordinate data collection, review data, analyse it and prepare a high-quality report.
Reports to Somalia MEAL Manager
Key working relations: Somalia MEAL Manager, Somalia Emergency Coordinator, Somalia Program Managers, Somalia Country Manager
Key Responsibilities
Develop an inception report, detailing the agreed upon study design, methodology, indicators, data- gathering tools, work plan schedule and budget to carry out the assignment, in consultation with CRS.
Conduct desk‐review of relevant project documents and secondary data on child protection
Develop quantitative and qualitative data gathering tools in consultation with CRS
Plan and coordinate quantitative and qualitative data collection
Conduct training for the data collection teams including pre‐testing of data collection tools
Work with the project management team in evaluation planning and logistics
Organize and facilitate team interaction
Provide support to evaluation team members to fulfil their obligations
Conduct Key Informant Interviews
Review, clean and analyse data collected
Write report on the findings and recommendations
Present preliminary findings to project stakeholders for validation
Incorporate input from project stakeholders and submit final report
Carry out or assist in additional duties assigned by the project staff Desired Qualifications and abilities
A minimum educational qualification of a master’s degree in Emergency response, Agriculture, Social Sciences, Economics or relevant field from recognized university
Must have a proven research experience in the Somalia context.
Has undertaken similar evaluations in the past 2 years in Somalia. This includes demonstrated ability to manage field procedures in the evaluation area.
Previous evaluation experience for a USAID project is an added advantage.
Solid experience in qualitative and quantitative studies.
Experienced in use of ICT4D solutions in data gathering.
Computer proficiency with good knowledge of MS office (Word, Excel, PowerPoint).
Excellent analytical and report writing skills
Excellent written and spoken English and Somali will be an added advantage
Excellent communication and interpersonal skills
Excellent time management skills
Ability to work promptly and accurately, and pay attention to detail
Ability to work well both independently and in a team
Available to be engaged during the entire survey period
Field Supervisors: Will take part in enumerator training, guide and supervise data collection
Report to Lead Consultant
Key responsibilities
Obtain sampling lists for each area in which his/her team will be working
Assign work to enumerators.
Maintain fieldwork control sheets and make sure assignments are carried out
Communicate any problems to the Lead Consultant and/or project staff
Try to develop a positive team spirit
Conduct regular spot-checks and re-interviews
Conduct regular review sessions with each enumerator
Receive data from enumerators (questionnaires, focus group guides etc.) at the end of each day
Produce a summary observation report detailing daily achievements, general observations, challenges and summary findings/ emerging themes.
Oversee entry of data into established data entry templates as necessary by team members at the end of each data collection day
Ensure that all evaluation procedures and protocols are followed
Carrying out or assist in additional duties assigned by the Lead Consultant Desired Qualifications and abilities
Familiarity with the Somalia context (Baidoa, Mogadishu and Afgooye districts)
Fluency in written and spoken Somali and English
Experienced in team management.
Familiarity with data collection on mobile technologies
Understands surveys ethics and protocols.
Understanding of data confidentiality issues.
Excellent verbal and written communication skills in English and Somali
Ability to work with minimum supervision
Excellent communication and interpersonal skills
Ability to multitask
Social Perceptiveness- Aware of other reactions and understands them
Ability to work quickly and accurately, and pay attention to detail
Ability to work well both independently and in a team ▪ Available to be engaged during the entire survey period.
Enumerators: Will administer questionnaires to respondent
Report to the Field Supervisors.
Key responsibilities
Locate households and identify respondents
Explain survey and/or focus group objectives and procedures to interviewees
Ask questions in accordance with instructions to obtain various specified information
Interpret questions to help interviewees’ comprehension
Identify and resolve inconsistencies in interviewees’ responses by means of appropriate questioning and/or explanation
Review data obtained from interview for completeness and accuracy.
Identify and report problems in obtaining valid data
Produce a daily observation report detailing daily achievements, general observations, challenges and summary findings/emerging themes.
Make and honor appointments made with respondents in cases where the respondent was not available for interview
Meet with supervisor daily to submit completed assignments and discuss progress
Carry out or assist in the additional procedures for data collection, as requested by the field supervisor Desired Qualifications and Abilities
Minimum secondary education
Prior experience conducting data collection for government programs and/or international NGOs in Somalia
Excellent verbal and written communication skills in English and Somali
Familiarity with mobile data collection technologies is preferred
Understands survey protocols and ethics.
Fluent in English and Somali language
Familiarity with Afgooye, Baidoa and Mogadishu districts geography
Must be in possession of a smartphone
Excellent communication and interpersonal skills
Social perceptiveness- Aware of other reactions and understands them
Excellent time management skills
Ability to work quickly and accurately, and pay attention to detail
Ability to work well both independently and in a team
Respectful and friendly in all interactions
Available to be engaged during the entire survey period
Role of CRS Role of Implementing Partners
Recruit external consultant for the evaluation
Provide consultant with background documents, reports and available secondary data for review
Validation of data collection tools
Make travel, accommodation and per-diem arrangements for consultant and team
Oversee the recruitment of enumerators by the consultant
Provide oversight for the data collection
Review and provide input on the consultant’s deliverables
Organize validation workshop for the presentation of preliminary findings to the project stakeholders
Ensure smooth flow of consultancy engagement processes including contractual obligations
Share the final evaluation report with all key stakeholders, including key project staff, partners, donor representative and government Provide sample frames from which respondents for the quantitative data will be drawn
Assist in the identification, contacting and locating of project beneficiaries and stakeholders
Support recruitment of enumerators
Conduct community sensitization and mobilization of respondents
Participate in the validation workshop

The composition of the evaluation team is flexible to change, depending on the approach of the consultant to address the limitations posed by the COVID-19 pandemic.
VI.REPORTING AND DISSEMINATION PLAN

VI.A. Final Evaluation Report
The consultant must submit two hard copies and a soft copy of the final evaluation report which is expected to be within 25 pages (without annexes) and with the following components:

Preliminary Pages (Title page, Table of Contents including a list of annexes, Acknowledgement, Executive Summary, List of Acronyms and Abbreviations, Definition of Terms and Concepts)
Introduction describing the project’s background and context, a description of the program, including the results framework and theory of change
Purpose and Objectives of Evaluation
Key evaluation questions (or objectives) and a statement of the scope of the evaluation
An overview of the evaluation approach, methodology and data sources, as well as limitations and delimitations of the evaluation itself
Evaluation findings, organized around the four key Evaluation Questions
Conclusions based on evaluation findings, organized around the four key Evaluation Questions
Lessons learned based on the evaluation findings
Recommendations based on evaluation findings, organized by audience: 1.) Donor/OFDA and 2.) CRS and its LIPs, as well as future Implementing Partners, etc.
Appendices (including all data collection tools, respondent lists, raw data collected, data analysis files, etc)

VI.B. Dissemination Plan

STAKEHOLDER/
AUDIENCE KEY FINDINGS CHANNEL(S) OF COMMUNICATION PRODUCT(S) TO SHARE
USAID/OFDA To establish the impact of the project components on the target beneficiaries.
Suggestions/recommendations with the potential to shape future programming. Email communication Final Evaluation Report
CRS and local implementing partners (LIPs) To establish the impact of the project component on the target beneficiaries.
Suggestions/recommendations with the potential to shape future programming
To assess and evaluate effectiveness of the response interventions in terms of its implementation approach and strategies Email communication
Dissemination and reflection meeting
Hard copies Final Evaluation Report and Appendices

VII. SCHEDULE AND LOGISTICS
The consultant should prepare a detailed workplan document, in which he/she describes the evaluation’s overall schedule (i.e., duration, phasing, timing) as well as work hours, required preparation work, conditions that might affect data collection, meeting-arranging procedures, and needed and available office space, cars, equipment, and local services (e.g., interviewers).
VIII. DELIVERABLES AND TIMELINE
The following items will be expected to be delivered during implementation, analysis, and reporting on the evaluation:
A detailed workplan, with target dates and deliverables identified and highlighted
All data collection tools
Sampling guidance and sample, as well as list of participants for KIIs
Cleaned quantitative dataset (for quantitative data collection methods)
Related codebooks, and data analysis files (Excel format required, other software syntax files optional)
Key informant interview forms/reports (MS word)
All photos and field notes with quotation from beneficiary verbatim
The final report shall be submitted in two (2) hard copies and one (1) soft copy.

DELIVERABLES ESTIMATED NUMBER OF DAYS NEEDED TO COMPLETE TARGET DATES TO COMPLETE
Initial meeting between CRS and the consultant to agree on the evaluation methodology and data collection tools 1 June 30, 2020
Consultant creates a detailed workplan as a deliverable which CRS will review and approve 1 July 2, 2020
Review of project documents and submission of an inception report with all data collection tools and guidance 2 July 3, 2020
Revision of the inception report and data collection tools by the consultant based on feedback provided by CRS 3 July 5, 2020
Translation of data collection tools 1 July 8, 2020
Coding of tools into the mobile data gathering platform. 1 July 9, 2020
Training for enumerators (including pretest; only if needed depending on the approach of the consultant on the COVID-19 situation) 3 July 10, 2020
Data collection (Qualitative and quantitative concurrently)
Translation of qualitative data. 7 July 13, 2020
Data cleaning and analysis with submission of data tables and syntax files and preliminary report. 6 July 20, 2020
Draft report submitted. 1 July 26, 2020
Results dissemination meeting. 1 July 27, 2020
Final report submitted to CRS with Datasets (or recordings and transcripts/notes), data file (in Excel), with all CRS feedback (on Draft Report) addressed 1 July 28, 2020
Total consultant engagement days (Estimated) 28 days

IX. BUDGET
CRS will provide for the following costs for the consultant team: payment for translators, data collectors, data processors (as needed), and secretarial services; equipment, etc. CRS will procure the consultant services based on a competitive rate and ability to complete the assignment. All costs are subject to change based on the approach of the consultant due to the ongoing COVID-19 pandemic.
X. ETHICAL CONSIDERATIONS
The below ethical considerations will be adhered to during the evaluation process:
The evaluation will be conducted by an independent and impartial external evaluator.
Quantitative data will be obtained from a randomly selected representative sample.
Participation in the survey will be voluntary.
Anonymity, confidentiality and safeguarding of survey data will be guaranteed.
There will be no risks and benefits for individual participants
The culture, norms and traditions of study populations will be respected.
Consent will be sought prior to commencing data gathering
Views and Opinions of the different survey subjects will be respected.
Abide by the “Do No Harm” principle especially in the context of COVID-19.
The consultant should limit the risk of spreading COVID-19 to communities and implementing partners staff by avoiding in-person data collection.
The consultant should always adhere to the COVID-19 directives issued by the Federal Government of Somalia and federal member states where the data collection is taking place.
Comply with USAID evaluation procedures by ensuring external consultant has been listed under Section 1.4.b.2.B of your award entitled “Sub-Award, Transfer, or Contracting Out of Any Work”
XX. Key evaluation compliance requirements
See the USAID evaluation policy (https://www.usaid.gov/evaluation/policy) and CRS – MEAL policies and procedures (available on request).
XXX. QUOTATION REQUIREMENTS
Interested applicants to send a technical and financial proposal for the work in line with the following guidance:
Capability statement: How the consultant or firm is structured for the assignment, the role each staff will play including the CVs of the key personnel who will take part in the consultancy
Technical Proposal: The consultant’s understanding and interpretation of the Terms of Reference (TOR), a detailed methodology and plan on how the data collection and analysis will be done and detailed implementation schedule for the evaluation.
Financial proposal: Itemized budget proposal that should include the consultancy fees and operational costs.
References: Names, addresses, telephone numbers of three organizations that you have conducted evaluations for within the last three years, that will act as professional referees
Evaluation reports: Final reports for the evaluations conducted for the three reference organizations provided