Terms of Reference
Second Midline Evaluation Consultancy Services for CARE Somalia’s Adolescent Girls’ Education in Somalia Project, funded by UK’s Girls’ Education Challenge and USAID
1. SUMMARY
CARE Somalia (henceforth referred to as “CARE”) is seeking to procure the services of a local or international evaluation company to manage the second midline evaluation of Adolescent Girls’ Education in Somalia (AGES), an initiative funded by the United Kingdom’s (UK) Foreign, Commonwealth and Development Office (FCDO) through its Girls’ Education Challenge- Leave No Girl Behind (GEC/ LNGB) initiative and by USAID. The GEC is a global initiative focusing on ensuring the expansion of education opportunities for marginalized girls at primary and secondary level, using rigorous evaluation practices. The AGES project seeks to improve learning and transition outcomes for 82,975 severely marginalised girls in conflict-affected areas of Banaadir, Hirshabelle, Jubaland and South West states of Somalia. AGES supports formal, non-formal and accelerated education for some of the most affected out-of-school girls, including internally displaced populations (IDPs), girls with disabilities, minorities, married and divorced girls, adolescent mothers, and those in child labour. AGES has high demands in terms of the quantity and quality of data to be collected to generate robust evidence which can be used by multiple stakeholders.
This tender includes two components:
CARE will accept bids inclusive of both components or only one of them.
The second midline evaluation will be an independent and rigorous study, which will enable the project to identify its effectiveness on achieving the expected outcomes and intermediate outcomes; assess to what extent the expected outcomes are being achieved by different subgroups of the population (equity); identify the extent to which different components of the intervention are contributing (or not) to the said outcomes and intermediate outcomes, testing the project’s Theory of Change (ToC); and assess delivery processes (efficiency and economy). The evaluation will ultimately assess if the AGES investment represents Value for Money (VfM) for the UK, US and Somali governments, while also identifying valuable lessons learned for girls’ education programming in fragile and conflict-affected contexts.
The second midline evaluation will use a pre-post design with a mixed methods approach. Quantitative data will be obtained from learning assessments conducted with two student cohorts (1 and 4) representative of treatment interventions; a household survey conducted with students’ families; a school survey; head counts; and classroom observations. Quantitative results will be triangulated with qualitative data obtained through focus group discussions, interviews, and participatory exercises with girls. The second midline evaluation will also include an analysis of contextual changes influencing outcomes, using secondary data collected by other development partners and monitoring data collected by the project during the course of the project. Findings from the contextual analysis will inform the interpretation of the second midline results and the preparation of recommendations based on those. The results of the second midline evaluation will be used to inform adaptations to project design and delivery modalities, seeking to maximise and equalise impact for all subgroups of girls targeted by the project.
This consultancy will also include a baseline evaluation for a representative sample of a third cohort of students enrolled by AGES in non-formal education classes. The baseline study will be conducted using a similar methodology to the second midline study and the findings will be included as a section of the second midline report.
2. BACKGROUND TO THE GEC PROGRAMME AND THE AGES PROJECT
2.1. GEC Programme Background
FCDO leads the UK’s work to end extreme poverty. FCDO is tackling the global challenges of our time including poverty and disease, mass migration, insecurity and conflict. FCDO’s work is building a safer, healthier, more prosperous world for people in developing countries and in the UK too.
FCDO is working to reach the Sustainable Development Goals (SDGs) by 2030. Progress on girls’ education is critical to the achievement of these targets. SDGs 4 and 5 specifically relate to education and achieving gender parity. SDG 4 specifically notes ‘inclusive and quality education for all and promote lifelong learning’.
USAID has joined efforts with FCDO’s Girls’ Education Challenge in its commitment to address barriers to the achievement of equitable education outcomes for all. In doing so, USAID contributes to the US Government’s Strategy on International Education, which commits American aid towards the achievement of SDG 4, expanding access to education for marginalised and vulnerable populations, such as children and youth affected by crises and conflict; improving learning outcomes and prepare girls for a productive future; and facilitating girls and women’s empowerment through gender-transformative educational opportunities. USAID’s partnership with FCDO’s GEC is aligned with the principles of its Education Policy, which prioritizes leveraging resources, promoting equity and inclusion, and focusing investments on improving learning and educational outcomes. In the specific case of AGES, it is expected that the programme will contribute to the expected outcomes of USAID Somalia’s Country Cooperation Development Strategy (CDCS) and USAID’s Youth in Development Policy.
Globally, 31 million primary age girls have never been to school[1]. The majority of these girls come from the poorest and most marginalised communities in the most disadvantaged locations, ethnic groups, etc.[2] Over the last 20 years, primary enrolments for girls have improved along with boys but completion rates are equally low for both sexes. At the secondary level, the differences between boys’ and girls’ participation rates really start to show. Significant disparities exist within countries, with the poorest girls from rural areas most severely subject to educational disadvantage – even at the primary level.[3] In the specific case of Somalia, such disparities are already observed in early primary grades, where the gross enrolment rate stands at 14%.[4] The GEC is helping the world’s poorest girls improve their lives through education and supporting better ways of getting girls in school and ensuring they receive a quality of education to transform their future.
PricewaterhouseCoopers LLP (PwC) and alliance partners have been contracted as the dedicated Fund Manager (FM) and is responsible for the day-to-day operation of the GEC. This includes establishing the recipient tendering process, supporting bidders, sifting and scoring proposals, monitoring Value for Money (VfM) and making project funding recommendations for FCDO approval. The FM also manages the relationships with the selected projects and oversees their Monitoring, Evaluation, and Learning operations.
Through the GEC, the UK government provided £355m between 2012 and 2017 to the FM to disburse to 37 individual projects across 18 countries across sub-Saharan Africa and South Asia to help girl’s education. In 2016 the GEC Transition window has been set up with additional UK funding to support the original GEC beneficiaries continue their journey through stages of education and further transit to upper grades and improve their learning[5]. In Somalia, the US government has contributed $15m to the GEC-funded AGES project to further expand its reach and impact.
CARE Somalia is implementing AGES, one of the GEC-funded projects. AGES’ intervention focuses on providing tailored, sustainable solutions to develop literacy, numeracy, and key life skills (financial literacy and knowledge of reproductive health) for 82,975 of the most marginalised girls and female youth in South Somalia, including girls with disabilities, those from minority groups, IDPs, married and divorced girls and adolescent mothers.
2.2. Project Background
Operating in Somalia since 1981, CARE currently works through three main programs: the first, CARE’s rural vulnerable women’s program, supports poor rural women and girls in addressing long term underlying causes of poverty and vulnerability, addressing social, economic, cultural and political obstacles to positive change. We help women and girls improve their economic status, access education and support them to play a greater role in local leadership and conflict resolution. Our second program area, the youth program, focuses on job creation and livelihood opportunities for poor youth through e.g. secondary education, vocational training, small business development, and microfinance. Thirdly, the emergency program provides direct humanitarian relief to victims of drought and conflict in Puntland, Mogadishu, and Lower Juba. CARE has its main offices in Hargeisa, Garowe, Mogadishu, and Erigavo, and satellite offices in Burao, El Afweyn, Kismayo and Bossaso. We work with a large number of local partners and maintain excellent relations with local governments.
Led by CARE in partnership with three local NGOs, AGES works with schools, communities, individual students, religious leaders, the Federal Ministry of Education, Culture and Higher Education (MOECHE) and State Ministries of Education (MoEs) in Hirshabelle, Jubaland and South-West, as well as the Banaadir Regional Administration to:
AGES is also providing the means for girls who are unable to attend school consistently to study remotely, through guided lessons and remote support; mobilising communities to follow up on dropouts and ensure their return to school; providing hygiene supplies to participants and schools; and supporting the implementation of psychosocial first aid through mentors, teachers, and peers.
The use of flexible learning tracks by the AGES project, complemented by remote education, allows girls engaged in labour, married and divorced girls, and pastoralist girls to attend learning sessions despite the limitations on their time. The work with religious leaders and Quranic teachers addresses the resistance to the inclusion of older and married girls and issues with their mobility and approval from family to attend classes, particularly in areas where recent influence from armed groups has resulted in severe curtailing of women’s freedom.
Physically disabled girls with limited mobility benefit from (1) access to ABE/ non-formal education courses close to home; (2) renovations to schools which include accessible features such as ramps; (3) access to remote learning allowing them to learn from home; (4) increased awareness of parents and community members of the importance of education for the disabled. Girls with special learning needs are benefitting from scholarships to attend special needs schools. Girls facing severe anxiety and depression are supported through the combination of psychosocial first aid and support networks through the GEFs.
Girls from minority clans and ethnic groups develop their self-confidence and aspirations and build connections beyond their own clans through participation in the GEFs. The participation in GEF networks will expand the engagement of the most marginalised adolescent girls and female youth in civic action. Furthermore, teachers have been trained to support girls who are lagging behind and first-generation learners, enhancing the likelihood of enrolment and retention in both ABE and formal education. Girls from displaced and ultra-poor families as well as orphaned children are overcoming financial barriers through the participation of parents, older adolescent girls, and female youth in VSL. Through the expansion of AGES activities funded by USAID, female youth also have the opportunity to further expand their economic empowerment through VSL networks and participation in skills training, as well as enhanced access to safety nets.
See Annex 1 for an overview of the project’s outcomes, intermediate outcomes, outputs, and associated indicators as well as the key activities under each output.
2.3. Overview of the project implementation timescales
Project start-date: September 7, 2018
Project end-date: August 31, 2024
2.4. Project beneficiaries
AGES seeks to support out-of-school adolescent girls and female youth ages 10-25 facing multiple barriers to enrolment, attendance, and learning. The project will enrol two cohorts of students in accelerated basic education (ABE), as well as five cohorts of students in formal school and non-formal/ life skills courses, reaching a total of 82,975 girls. The first cohort enrolled by the project included 20,468 girls[6]. The pre-enrolment assessment of Cohort 1 students indicated that 31% are IDPs; 16% belong to marginalised minority groups; 40% are Af-Maay speakers (including 18% who are Af-Maay speakers in areas where this is a minority language); 13% are orphans; 10% do not live with parents; 4% are married and 4% are divorced. The enrolment of girls with disabilities was prioritised by the project, with extensive sensitisation of stakeholders and multiple rounds of identification being conducted in order to maximise the likelihood of enrolment. A total of 253 Cohort 1 girls have disabilities other than mental health issues, while 6,140 girls were estimated to face severe anxiety and/or depression. The following cohorts included progressively larger proportions of minority students and IDPs, reaching 27% minority girls and 43% IDPs in NFE cohort 4. A breakdown of the out-of-school girls enrolled to date by cohort is provided on Table 1 below.
Intervention pathway
Which girls follow this pathway?
Cohort 1 enrolment
Cohort 2 enrolment
Cohort 3
Cohort 4
Cohort 5
How long will the intervention last?
How many cohorts are there?
What literacy and numeracy levels are the girls starting at?
What does success look like for learning?
What does success look like for Transition?
Formal school
Girls aged 10-12
6,623
5,594
6,479
To be enrolled
4 years
3
Grade 0-1 for literacy-numeracy
Girls achieve grade 4 level for numeracy
Girls enrolled and retained in formal school
Accelerated basic education (ABE)
Girls aged 13-16
7,241
6,035
2 years
2
Grade 0-1 for literacy-numeracy (original estimate; baseline results show otherwise)
Girls achieve grade 4 level for numeracy
Girls enrolled and retained in ABE; a proportion of the girls transition into formal school upon completion
NFE/Life skills
Girls aged 15-25 (17-19 in Cohorts 1 and 2; 15-25 in subsequent cohorts[7])
6,604
4,319
4,198
13,443
To be enrolled
11 months
5
Grade 0-1 for literacy-numeracy (original estimate; baseline results show otherwise)
Girls are able to read with comprehension and achieve mastery of basic operations (addition and subtraction with problem-solving)
Girls enrolled and completing life skills training; transition into higher levels of education, TVET or employment/ self-employment
Table 1 – Breakdown of participants by cohort
2.5. Approach during the COVID-19 pandemic
Following the closure of schools in March – August 2020, movement restrictions and a severe economic crisis, the project team and the Fund Manager have agreed on a set of midterm adaptations to mitigate the impact of the pandemic on the students and schools supported by AGES. Those include:
-Social protection and safety: Support to parents engaged in savings groups, including coaching of adult and adolescent savings groups; linkages with banks; provision of seed funding; provision of hygiene materials to families; training of community health workers to support GEFs;
-Continuation of teaching and learning: Remote learning content provided to students; strengthening supervision and professional support to teachers by regional and district education officers; printing and dissemination of ABE curriculum materials; provision of teaching and learning materials to students in ABE and formal schools;
– Return to school and learning centres: Follow-up on dropout cases; provision of PPEs and hygiene supplies to schools; cleaning of schools prior to reopening; coordination with other development partners on provision of WASH supplies;
-Wellbeing: Provision of grants to GEFs to implement girl-led activities; awareness raising on COVID-19 prevention; provision of psychosocial support to children;
-Combatting exclusionary norms: Community mobilisation to support remote learning; mobilisation of religious leaders to promote equity in education; strengthen official tracking of out-of-school children and gender-sensitive and inclusive quality assurance processes.
Several adaptations made in response to the pandemic continue to be applied to date in response to other shocks, namely the ongoing drought (2021-2022) and the escalation of violence in parts of the country, which have contributed to school closures, displacement, and restrictions to girls’ movement. Continuing adaptations include support to remote and remedial learning; early warning and dropout prevention systems; provision of water to schools; psychosocial first aid; and integration with social protection systems.
3.1. Rationale for the Evaluation
The findings from the second midline evaluation of the first cohort (of all three education pathways) and the fourth cohort of NFE will be primarily used:
In addition, findings from the baseline evaluation of the fifth cohort of NFE students will be used to establish baseline values for this cohort, enabling the project to determine progress at the final evaluation. The baseline for this cohort will also allow the project to compare the trajectories of this and other cohorts at the final evaluation and better understand how they differ, particularly considering the interruption of project activities experienced by the first cohort and the major shocks (drought and conflict) faced by the fourth cohort. The baseline evaluation of the fifth cohort of NFE students will also identify the factors affecting the expected outputs and intermediate outcomes of the expanded activities with older adolescent girls and female youth participating in cohorts 4 and 5 of the NFE component, assessing the validity of the project’s revised Theory of Change and informing adaptations of those subcomponents of the AGES intervention.
The project is seeking to procure the services of an independent external evaluation company to conduct a mixed-methods, gender-sensitive second midline study that is inclusive of persons with disabilities and other marginalised sub-groups within the targeted population (pastoralists, minority clans, displaced girls, girls affected by early and forced marriage, girls in child labour). The second midline study will assess the economy and efficiency of delivery, effectiveness, and equity of the intervention in achieving its outcomes, thus determining VfM, and establish the results of the project at outcome, intermediate outcome and impact levels. It will also conduct an extensive exploratory analysis and review of secondary data to identify the factors affecting the project’s expected outcomes. The study will also include a baseline for a newly enrolled NFE cohort, which will determine baseline values for future assessment of progress against expected outcomes and comparison of results with those of prior cohorts at the final evaluation.
The following evaluation questions have been agreed with the GEC Fund Manager. An additional set of questions has been added to enable the project to understand the combined impact of the ongoing crises on the students and targeted schools. Additional questions may be added by the external evaluator as appropriate.
-Likelihood of continuity of interventions, particularly for ABE and NFE;
-Policy shifts in domains of change (gender in education; safeguarding; inclusive and special needs education; access to alternative/non-formal education)
– Leveraged investment in access/ gender
-Sector capacity building
-Social norm changes
The following evaluation questions refer only to the NFE cohorts 4 and 5.
The project uses a longitudinal, mixed-methods pre-post evaluation design with additionality being measured through the use of benchmarks for performance, set for each type of education intervention, corresponding to: an improvement of 0.2 standard deviation over and above the following grade level for formal education; 0.2 standard deviation over and above the corresponding grade for ABE (in relation to the formal grade performance); and minimum achievement levels for literacy and numeracy for at least 70% of the students attending NFE/life skills classes.
The following methods will be used:
The girl module includes questions on her educational experience; disability; leadership skills (Youth Leadership Index questions/ YLI, participation in GEFs and civic action); menstrual hygiene management; safety and security; COVID-19 (preventative measures and impact); exposure to interventions (GEF, savings); addressing mental health issues; engagement in economic empowerment activities and IGAs, and limiting/enabling factors linked to those; participation in governance processes; and access to safety nets. Some questions of the girl module will apply only to older adolescent girls and female youth enrolled in NFE courses. The girl module will be answered by all girls in the sample (i.e. cohorts 1, 4 and 5).
School-level tools
Qualitative data will be collected from 12 sample sites. All qualitative data will be recorded, integrally transcribed, and translated.
Quantitative data will be collected using electronic tools.
All tools will be piloted prior to data collection and refined as necessary (with exception of the learning assessments, which will be piloted and calibrated in advance by CARE). The external evaluator will be responsible for piloting the tools, while CARE will liaise with the Fund Manager and the MOECHE for approval.
Potential adaptations in response to COVID-19 and security restrictions
Current conditions allow for in-person data collection in South Somalia, provided enumeration teams adhere strictly to safety protocols for their own safety and the safety of project participants and other stakeholders at community level. CARE does recognise, however, the potential for rapid shifts in field conditions and the need to plan for alternative scenarios in case of a dramatic escalation in infection rates and/or movement restrictions due to other reasons, including instability, conflict, and natural disasters. In case those scenarios occur, CARE will work with the external evaluation and data collection company / companies and the Fund Manager to agree on adaptations, including phone-based data collection with abridged tools and modified versions of learning assessments.
The sampling process is described above and in the project’s MEL Framework. To allow for a more streamlined procedure, increase comparability of results, and avoid extraordinarily high evaluation costs, the cohort 5 baseline will be conducted in the same sites (i.e. learning centres) as cohort 4. Those sites were selected in 2021 using stratified random sampling (proportional to the distribution of the cohort 4 population). The project will provide a list of all enrolled girls in those learning centres (sampling framework) to the external evaluator, who will be required to help finalise and operationalise the sampling approach for both qualitative and quantitative samples. In each sample site, students will be randomly selected from the list of enrolled out-of-school girls. Please refer to section 6.5 above for the required sample sizes. The final sample should be representative of the fifth NFE cohort and collected in an appropriate manner, to ensure:
The evaluation must consider the safety of participants and especially children and vulnerable individuals at all stages. The selected evaluation company will need to demonstrate how they have considered the protection of children through the different data collection stages, including recruitment and training of research staff, data collection, data analysis, and report writing.
The selected evaluation company is required to set out their approach to ensuring complete compliance with international good practice with regards to research ethics and protocols, and in particular to safeguarding children, vulnerable groups (including people with disabilities) and those in fragile and conflict-affected states. Consideration should be given to:
Parental consent concerning data collection from children or collation of data about children;
Risk management plan: It is important that the successful bidder has taken all reasonable measures to mitigate any potential risk to the delivery of the required outputs for this study. Therefore, the selected evaluation company should submit a comprehensive risk management plan covering:
Quality assurance plan: the selected evaluation company is required to submit a quality assurance plan that sets out the systems and processes for quality assuring the research process and deliverables from start to finish of the project. This plan should include the proposed approaches to:
In the first instance, the selected evaluation company should refer to the GEC website:
https://www.gov.uk/guidance/girls-education-challenge for general information concerning the Girls’ Education Challenge.
The selected evaluation company should refer to the following GEC programme documentation:
The selected evaluation company should refer to the following GEC project documentation that includes:
The selected evaluation company should also refer to relevant country data and information that is currently available, as required, to prepare the proposal.
4.1. Professional Skills and Qualifications
Qualifications: the selected evaluation company is required to clearly identify and provide CVs for all those proposed in the Evaluation Team, clearly stating their roles and responsibilities for this study. Bidders are required to provide CVs by component, i.e. (1) data collection and (2) evaluation design, analysis, and reporting. In case the same team is proposed for both components, bidders should indicate that this is the case.
The proposed evaluation team should include the technical expertise and practical experience required to deliver the scope of work, with regards to:
Component 1: Data collection
Component 2: Evaluation design, data analysis and reporting
The day–to–day project management of this study will be under the responsibility of Paul Odhiambo, M&E and Knowledge Manager, AGES project/ CARE Somalia.
4.2. Deliverables and Schedule
Expected Tasks
Component 1: Data collection
Component 2: Evaluation design, data analysis and reporting
1. Review the project’s MEL framework, other relevant project documents and external literature related to the Somali context and in particular, the education sector in country;
2. Submit an inception report (draft and final), that outlines the research methodology, including data collection tools, detailed analysis framework and detailed work plan outlining all tasks to be completed by each of the members of the study team;
3. Finalization of tools, including adaptations post-piloting and back-translation supervision;
4. Technical support for enumerator training content and toolkits (qualitative and quantitative) and supervise/monitor training on data collection tools and their application;
5. Follow up on and directly supervise enumerators’ work in the field during data collection (qualitative and quantitative) and provide the necessary technical support to the data collection team to ensure quality of data, including overnight checks of electronic datasets;
6. Validate all datasets, working in coordination with the data collection company, and collate data as necessary for analysis;
7. Analyse the collected data in accordance with the agreed analysis framework.
8. Prepare a comprehensive draft and final study report, according to the format provided by the Fund Manager, and inclusive of the GEC Outcomes Spreadsheet if required.
Deliverables
In reference to the scope of work above, the consultant team is expected to accomplish and submit the following:
Component 1: Data collection
Component 2: Evaluation design, data analysis and reporting
Project milestones: bidders are required to include in their detailed work plans the milestones set out below.
Component 1: Data collection
Milestone
Timeframe/ number of days
Training outline and materials
Draft Inception Report submitted for review and comments
Review of inception report completed and comments returned to supplier/consultant
Final Inception Report submitted
Electronic versions of all tools, inclusive of translation
Training
Tool piloting
Data collection starts
Weekly progress reports
Data collection completed
Draft datasets submitted
Clean datasets submitted
Data collection report submitted
Component 2: Evaluation design, data analysis and reporting
Milestone
Timeframe/number of days
Literature/document review / review of project’s theory of change, impact logic
Draft Inception Report submitted for review and comments
Presentation to Evaluation Steering Group
Review of inception report completed and comments returned to supplier/consultant
Final Inception Report submitted
Training
Tool piloting
Study starts
Study completed
Data analysis and drafting of the report
Draft Study Report submitted for review
Presentation to Evaluation Steering Group
Review by Project Management and stakeholders completed / comments provided to Supplier/Consultant
Supplier/Consultant addresses comments and revises the study
Final report submitted
The data collection and independent evaluation company/ companies will be expected to identify a Project Director and Project Manager for communication and reporting purposes. At the inception meeting, the Project Manager (s) will be expected to submit a full contact list of all those involved in the second midline evaluation.
The selected bidder (s) will be expected to report to the Evaluation Steering Group and attend all meetings as agreed with the Project Evaluation Manager. The successful bidder (s) will be required to submit to the Project Evaluation Manager weekly progress reports (by email) during the study periods summarising activities /tasks completed to date (percent achieved), time spent etc.
In addition to the Evaluation Steering Group, the Ministries of Education will also be part of the study. Regional/State MoE staff will liaise with the sampled schools to ensure their support to the study, working closely with CARE and Consortium Partners. No MOECHE/ MOE staff will participate in the actual data collection exercise to ensure an independent process. The MOECHE research coordinator and gender focal person will provide input to the draft evaluation tools and report.
The deadline for submission of bids is 17th December 2022. Interested bidders are welcome to request for the Project Logframe (Annex 1) or submit questions until 13th December 2022. To Paul.Odhiambo@care.org.
Annex 1: Project Logframe
Annex 2: Risk Management Template
Risk type
Risk
Likelihood
(1-5, 1 – not likely, 5 – highly likely)
Severity
(minor; moderate; major or severe)
Mitigation
(please briefly outline the measures you propose to mitigate these risks)
How to Apply.
All applications MUST be accompanied by a technical and financial proposal.
Interested consultants or firms are expected to submit their applications to: SOM.Consultant@care.org .Please **indicate “**Second Midline Evaluation Consultancy Services for AGES project” as the subject heading not later than 15, December 2022.
Female applicants with requisite Somalia experience are highly encouraged.
CARE is an equal opportunity employer promoting gender, equity, and diversity. Female candidates are strongly encouraged to apply. Our selection process reflects our commitment to the protection of children from abuse
All applications MUST be accompanied by a technical and financial proposal.
Interested consultants or firms are expected to submit their applications to: SOM.Consultant@care.org .Please **indicate “**Second Midline Evaluation Consultancy Services for AGES project” as the subject heading not later than 15, December 2022.
Female applicants with requisite Somalia experience are highly encouraged.
CARE is an equal opportunity employer promoting gender, equity, and diversity. Female candidates are strongly encouraged to apply. Our selection process reflects our commitment to the protection of children from abuse