Second Midline Evaluation Consultancy Services for AGES project

  • Location:
  • Salary:
    negotiable
  • Job type:
    Contract
  • Posted:
    10 months ago
  • Category:
    Monitoring and Evaluation
  • Deadline:
    15/11/2022

Terms of Reference

Second Midline Evaluation Consultancy Services for CARE Somalia’s Adolescent Girls’ Education in Somalia Project, funded by UK’s Girls’ Education Challenge and USAID

1. SUMMARY

CARE Somalia (henceforth referred to as “CARE”) is seeking to procure the services of a local or international evaluation company to manage the second midline evaluation of Adolescent Girls’ Education in Somalia (AGES), an initiative funded by the United Kingdom’s (UK) Foreign, Commonwealth and Development Office (FCDO) through its Girls’ Education Challenge- Leave No Girl Behind (GEC/ LNGB) initiative and by USAID. The GEC is a global initiative focusing on ensuring the expansion of education opportunities for marginalized girls at primary and secondary level, using rigorous evaluation practices. The AGES project seeks to improve learning and transition outcomes for 82,975 severely marginalised girls in conflict-affected areas of Banaadir, Hirshabelle, Jubaland and South West states of Somalia. AGES supports formal, non-formal and accelerated education for some of the most affected out-of-school girls, including internally displaced populations (IDPs), girls with disabilities, minorities, married and divorced girls, adolescent mothers, and those in child labour. AGES has high demands in terms of the quantity and quality of data to be collected to generate robust evidence which can be used by multiple stakeholders.

This tender includes two components:

  1. Data collection, including: Training of enumerators; data collection; quality assurance of fieldwork; data cleaning and collation; and regular reporting to CARE;
  2. Evaluation design, data analysis and reporting, including: Modifications to baseline and 1st midline quantitative tools; design of qualitative tools; design of the analysis framework; technical guidance to the data collection company on data collection methodologies, in case the two components are awarded to separate companies; data analysis; reporting; and preparation of dissemination products.

CARE will accept bids inclusive of both components or only one of them.

The second midline evaluation will be an independent and rigorous study, which will enable the project to identify its effectiveness on achieving the expected outcomes and intermediate outcomes; assess to what extent the expected outcomes are being achieved by different subgroups of the population (equity); identify the extent to which different components of the intervention are contributing (or not) to the said outcomes and intermediate outcomes, testing the project’s Theory of Change (ToC); and assess delivery processes (efficiency and economy). The evaluation will ultimately assess if the AGES investment represents Value for Money (VfM) for the UK, US and Somali governments, while also identifying valuable lessons learned for girls’ education programming in fragile and conflict-affected contexts.

The second midline evaluation will use a pre-post design with a mixed methods approach. Quantitative data will be obtained from learning assessments conducted with two student cohorts (1 and 4) representative of treatment interventions; a household survey conducted with students’ families; a school survey; head counts; and classroom observations. Quantitative results will be triangulated with qualitative data obtained through focus group discussions, interviews, and participatory exercises with girls. The second midline evaluation will also include an analysis of contextual changes influencing outcomes, using secondary data collected by other development partners and monitoring data collected by the project during the course of the project. Findings from the contextual analysis will inform the interpretation of the second midline results and the preparation of recommendations based on those. The results of the second midline evaluation will be used to inform adaptations to project design and delivery modalities, seeking to maximise and equalise impact for all subgroups of girls targeted by the project.

This consultancy will also include a baseline evaluation for a representative sample of a third cohort of students enrolled by AGES in non-formal education classes. The baseline study will be conducted using a similar methodology to the second midline study and the findings will be included as a section of the second midline report.

2. BACKGROUND TO THE GEC PROGRAMME AND THE AGES PROJECT

2.1. GEC Programme Background

FCDO leads the UK’s work to end extreme poverty. FCDO is tackling the global challenges of our time including poverty and disease, mass migration, insecurity and conflict. FCDO’s work is building a safer, healthier, more prosperous world for people in developing countries and in the UK too.

FCDO is working to reach the Sustainable Development Goals (SDGs) by 2030. Progress on girls’ education is critical to the achievement of these targets. SDGs 4 and 5 specifically relate to education and achieving gender parity. SDG 4 specifically notes ‘inclusive and quality education for all and promote lifelong learning’.

USAID has joined efforts with FCDO’s Girls’ Education Challenge in its commitment to address barriers to the achievement of equitable education outcomes for all. In doing so, USAID contributes to the US Government’s Strategy on International Education, which commits American aid towards the achievement of SDG 4, expanding access to education for marginalised and vulnerable populations, such as children and youth affected by crises and conflict; improving learning outcomes and prepare girls for a productive future; and facilitating girls and women’s empowerment through gender-transformative educational opportunities. USAID’s partnership with FCDO’s GEC is aligned with the principles of its Education Policy, which prioritizes leveraging resources, promoting equity and inclusion, and focusing investments on improving learning and educational outcomes. In the specific case of AGES, it is expected that the programme will contribute to the expected outcomes of USAID Somalia’s Country Cooperation Development Strategy (CDCS) and USAID’s Youth in Development Policy.

Globally, 31 million primary age girls have never been to school[1]. The majority of these girls come from the poorest and most marginalised communities in the most disadvantaged locations, ethnic groups, etc.[2] Over the last 20 years, primary enrolments for girls have improved along with boys but completion rates are equally low for both sexes. At the secondary level, the differences between boys’ and girls’ participation rates really start to show. Significant disparities exist within countries, with the poorest girls from rural areas most severely subject to educational disadvantage – even at the primary level.[3] In the specific case of Somalia, such disparities are already observed in early primary grades, where the gross enrolment rate stands at 14%.[4] The GEC is helping the world’s poorest girls improve their lives through education and supporting better ways of getting girls in school and ensuring they receive a quality of education to transform their future.

PricewaterhouseCoopers LLP (PwC) and alliance partners have been contracted as the dedicated Fund Manager (FM) and is responsible for the day-to-day operation of the GEC. This includes establishing the recipient tendering process, supporting bidders, sifting and scoring proposals, monitoring Value for Money (VfM) and making project funding recommendations for FCDO approval. The FM also manages the relationships with the selected projects and oversees their Monitoring, Evaluation, and Learning operations.

Through the GEC, the UK government provided £355m between 2012 and 2017 to the FM to disburse to 37 individual projects across 18 countries across sub-Saharan Africa and South Asia to help girl’s education. In 2016 the GEC Transition window has been set up with additional UK funding to support the original GEC beneficiaries continue their journey through stages of education and further transit to upper grades and improve their learning[5]. In Somalia, the US government has contributed $15m to the GEC-funded AGES project to further expand its reach and impact.

CARE Somalia is implementing AGES, one of the GEC-funded projects. AGES’ intervention focuses on providing tailored, sustainable solutions to develop literacy, numeracy, and key life skills (financial literacy and knowledge of reproductive health) for 82,975 of the most marginalised girls and female youth in South Somalia, including girls with disabilities, those from minority groups, IDPs, married and divorced girls and adolescent mothers.

2.2. Project Background

Operating in Somalia since 1981, CARE currently works through three main programs: the first, CARE’s rural vulnerable women’s program, supports poor rural women and girls in addressing long term underlying causes of poverty and vulnerability, addressing social, economic, cultural and political obstacles to positive change. We help women and girls improve their economic status, access education and support them to play a greater role in local leadership and conflict resolution. Our second program area, the youth program, focuses on job creation and livelihood opportunities for poor youth through e.g. secondary education, vocational training, small business development, and microfinance. Thirdly, the emergency program provides direct humanitarian relief to victims of drought and conflict in Puntland, Mogadishu, and Lower Juba. CARE has its main offices in Hargeisa, Garowe, Mogadishu, and Erigavo, and satellite offices in Burao, El Afweyn, Kismayo and Bossaso. We work with a large number of local partners and maintain excellent relations with local governments.

Led by CARE in partnership with three local NGOs, AGES works with schools, communities, individual students, religious leaders, the Federal Ministry of Education, Culture and Higher Education (MOECHE) and State Ministries of Education (MoEs) in Hirshabelle, Jubaland and South-West, as well as the Banaadir Regional Administration to:

  1. Increase the availability of flexible, quality learning opportunitiestailored to the needs of ultra-marginalised out-of-school girls:
  • Accelerated basic education (ABE) including compressed primary content;
  • Increased access to quality formal education through agreements with education providers and teacher training (reading, numeracy, child protection and gender-equitable practices);
  • Non-formal education (NFE/life skills course) for older girls (ages 15-25), teaching literacy, numeracy, financial literacy, savings, and basic business planning.
  1. Facilitate a social norm change movement towards broader life opportunities for adolescent girls and boys:
  • Religious leaders trained on Islamic principles to support gender equity and education for girls;
  • Community leaders and Community Education Committees (CECs) sensitised and mobilised to work with parents to identify and support extremely marginalised girls to enrol in learning opportunities and attend classes regularly;
  • Mothers and mothers-in-law mobilised through village savings and loans associations (VSLAs) and enrolment drives;
  • Girls trained on leadership skills, menstrual hygiene management, savings and forming peer support networks at Girls’ Empowerment Forums (GEFs);
  • Boys mentored through Boys’ Empowerment Forums (BEFs);
  • Messages reinforced via radio and social media.
  1. Build institutional capacity: Support to strengthen monitoring of education services; develop decentralised management capacity; strengthen mainstreaming of inclusive approaches and safeguarding mechanisms; and increase technical capacity for planning and management of non-formal education services.
  2. Facilitate the engagement of girls and female youth in collective action to claim rights through support to interventions led by GEFs and GEF networks.
  3. Support increased access of female youth to other development opportunities, including employability skills through technical-vocational education and training (TVET); economic empowerment opportunities; and participatory accountability processes.
  4. Increase girls and female youth’s access to critical services, such as legal services, governance institutions and safety nets through humanitarian support.

AGES is also providing the means for girls who are unable to attend school consistently to study remotely, through guided lessons and remote support; mobilising communities to follow up on dropouts and ensure their return to school; providing hygiene supplies to participants and schools; and supporting the implementation of psychosocial first aid through mentors, teachers, and peers.

The use of flexible learning tracks by the AGES project, complemented by remote education, allows girls engaged in labour, married and divorced girls, and pastoralist girls to attend learning sessions despite the limitations on their time. The work with religious leaders and Quranic teachers addresses the resistance to the inclusion of older and married girls and issues with their mobility and approval from family to attend classes, particularly in areas where recent influence from armed groups has resulted in severe curtailing of women’s freedom.

Physically disabled girls with limited mobility benefit from (1) access to ABE/ non-formal education courses close to home; (2) renovations to schools which include accessible features such as ramps; (3) access to remote learning allowing them to learn from home; (4) increased awareness of parents and community members of the importance of education for the disabled. Girls with special learning needs are benefitting from scholarships to attend special needs schools. Girls facing severe anxiety and depression are supported through the combination of psychosocial first aid and support networks through the GEFs.

Girls from minority clans and ethnic groups develop their self-confidence and aspirations and build connections beyond their own clans through participation in the GEFs. The participation in GEF networks will expand the engagement of the most marginalised adolescent girls and female youth in civic action. Furthermore, teachers have been trained to support girls who are lagging behind and first-generation learners, enhancing the likelihood of enrolment and retention in both ABE and formal education. Girls from displaced and ultra-poor families as well as orphaned children are overcoming financial barriers through the participation of parents, older adolescent girls, and female youth in VSL. Through the expansion of AGES activities funded by USAID, female youth also have the opportunity to further expand their economic empowerment through VSL networks and participation in skills training, as well as enhanced access to safety nets.

See Annex 1 for an overview of the project’s outcomes, intermediate outcomes, outputs, and associated indicators as well as the key activities under each output.

2.3. Overview of the project implementation timescales

Project start-date: September 7, 2018

Project end-date: August 31, 2024

2.4. Project beneficiaries

AGES seeks to support out-of-school adolescent girls and female youth ages 10-25 facing multiple barriers to enrolment, attendance, and learning. The project will enrol two cohorts of students in accelerated basic education (ABE), as well as five cohorts of students in formal school and non-formal/ life skills courses, reaching a total of 82,975 girls. The first cohort enrolled by the project included 20,468 girls[6]. The pre-enrolment assessment of Cohort 1 students indicated that 31% are IDPs; 16% belong to marginalised minority groups; 40% are Af-Maay speakers (including 18% who are Af-Maay speakers in areas where this is a minority language); 13% are orphans; 10% do not live with parents; 4% are married and 4% are divorced. The enrolment of girls with disabilities was prioritised by the project, with extensive sensitisation of stakeholders and multiple rounds of identification being conducted in order to maximise the likelihood of enrolment. A total of 253 Cohort 1 girls have disabilities other than mental health issues, while 6,140 girls were estimated to face severe anxiety and/or depression. The following cohorts included progressively larger proportions of minority students and IDPs, reaching 27% minority girls and 43% IDPs in NFE cohort 4. A breakdown of the out-of-school girls enrolled to date by cohort is provided on Table 1 below.

Intervention pathway

Which girls follow this pathway?

Cohort 1 enrolment

Cohort 2 enrolment

Cohort 3

Cohort 4

Cohort 5

How long will the intervention last?

How many cohorts are there?

What literacy and numeracy levels are the girls starting at?

What does success look like for learning?

What does success look like for Transition?

Formal school

Girls aged 10-12

6,623

5,594

6,479

To be enrolled

4 years

3

Grade 0-1 for literacy-numeracy

Girls achieve grade 4 level for numeracy

Girls enrolled and retained in formal school

Accelerated basic education (ABE)

Girls aged 13-16

7,241

6,035

2 years

2

Grade 0-1 for literacy-numeracy (original estimate; baseline results show otherwise)

Girls achieve grade 4 level for numeracy

Girls enrolled and retained in ABE; a proportion of the girls transition into formal school upon completion

NFE/Life skills

Girls aged 15-25 (17-19 in Cohorts 1 and 2; 15-25 in subsequent cohorts[7])

6,604

4,319

4,198

13,443

To be enrolled

11 months

5

Grade 0-1 for literacy-numeracy (original estimate; baseline results show otherwise)

Girls are able to read with comprehension and achieve mastery of basic operations (addition and subtraction with problem-solving)

Girls enrolled and completing life skills training; transition into higher levels of education, TVET or employment/ self-employment

Table 1 – Breakdown of participants by cohort

2.5. Approach during the COVID-19 pandemic

Following the closure of schools in March – August 2020, movement restrictions and a severe economic crisis, the project team and the Fund Manager have agreed on a set of midterm adaptations to mitigate the impact of the pandemic on the students and schools supported by AGES. Those include:

-Social protection and safety: Support to parents engaged in savings groups, including coaching of adult and adolescent savings groups; linkages with banks; provision of seed funding; provision of hygiene materials to families; training of community health workers to support GEFs;

-Continuation of teaching and learning: Remote learning content provided to students; strengthening supervision and professional support to teachers by regional and district education officers; printing and dissemination of ABE curriculum materials; provision of teaching and learning materials to students in ABE and formal schools;

– Return to school and learning centres: Follow-up on dropout cases; provision of PPEs and hygiene supplies to schools; cleaning of schools prior to reopening; coordination with other development partners on provision of WASH supplies;

-Wellbeing: Provision of grants to GEFs to implement girl-led activities; awareness raising on COVID-19 prevention; provision of psychosocial support to children;

-Combatting exclusionary norms: Community mobilisation to support remote learning; mobilisation of religious leaders to promote equity in education; strengthen official tracking of out-of-school children and gender-sensitive and inclusive quality assurance processes.

Several adaptations made in response to the pandemic continue to be applied to date in response to other shocks, namely the ongoing drought (2021-2022) and the escalation of violence in parts of the country, which have contributed to school closures, displacement, and restrictions to girls’ movement. Continuing adaptations include support to remote and remedial learning; early warning and dropout prevention systems; provision of water to schools; psychosocial first aid; and integration with social protection systems.

  1. CONSULTANCY PURPOSE AND DESIGN

3.1. Rationale for the Evaluation

The findings from the second midline evaluation of the first cohort (of all three education pathways) and the fourth cohort of NFE will be primarily used:

  • To determine the achievement of expected outcomes against benchmarks established at baseline and progress against intermediate outcome targets;
  • To identify factors affecting expected outcomes and if/how these map out against the intermediate outcomes and outputs selected by the project, thus testing its Theory of Change (ToC);
  • To gain up to date understanding of the project context to inform ongoing project strategy;
  • By the project management team, project partners and stakeholders to inform improvements in the delivery of the project during its lifetime;
  • To demonstrate accountability for the funding received to FCDO, other UK and US Government Departments, UK and US taxpayers, UK and US media;
  • By the project management team to leverage additional resources from existing and new partners and stakeholders in order to scale-up and sustain the activities /benefits delivered by the project;
  • By the project management team to support the on-going development and implementation of the project’s sustainability and succession strategies;
  • By partners, stakeholders and the Government to learn lessons from the project for the purpose of informing education programming and sector planning in country;
  • By the Fund Manager to feed into and identify insights in order to inform programme level questions; and
  • By other donors, academic institutions, and education networks to inform the wider policy debate concerning the education of girls and marginalised girls.

In addition, findings from the baseline evaluation of the fifth cohort of NFE students will be used to establish baseline values for this cohort, enabling the project to determine progress at the final evaluation. The baseline for this cohort will also allow the project to compare the trajectories of this and other cohorts at the final evaluation and better understand how they differ, particularly considering the interruption of project activities experienced by the first cohort and the major shocks (drought and conflict) faced by the fourth cohort. The baseline evaluation of the fifth cohort of NFE students will also identify the factors affecting the expected outputs and intermediate outcomes of the expanded activities with older adolescent girls and female youth participating in cohorts 4 and 5 of the NFE component, assessing the validity of the project’s revised Theory of Change and informing adaptations of those subcomponents of the AGES intervention.

    1. Objectives of this Consultancy

The project is seeking to procure the services of an independent external evaluation company to conduct a mixed-methods, gender-sensitive second midline study that is inclusive of persons with disabilities and other marginalised sub-groups within the targeted population (pastoralists, minority clans, displaced girls, girls affected by early and forced marriage, girls in child labour). The second midline study will assess the economy and efficiency of delivery, effectiveness, and equity of the intervention in achieving its outcomes, thus determining VfM, and establish the results of the project at outcome, intermediate outcome and impact levels. It will also conduct an extensive exploratory analysis and review of secondary data to identify the factors affecting the project’s expected outcomes. The study will also include a baseline for a newly enrolled NFE cohort, which will determine baseline values for future assessment of progress against expected outcomes and comparison of results with those of prior cohorts at the final evaluation.

    1. Evaluation Questions

The following evaluation questions have been agreed with the GEC Fund Manager. An additional set of questions has been added to enable the project to understand the combined impact of the ongoing crises on the students and targeted schools. Additional questions may be added by the external evaluator as appropriate.

  1. What impact did AGES have on the transition of highly marginalised girls into education/learning/training or work opportunities?
  2. What works to facilitate transition of highly marginalised girls into education / training / employment and to increase learning?
  3. How sustainable were the activities funded by the GEC and was the programme successful in leveraging additional interest, investment, and policy change?

-Likelihood of continuity of interventions, particularly for ABE and NFE;

-Policy shifts in domains of change (gender in education; safeguarding; inclusive and special needs education; access to alternative/non-formal education)

– Leveraged investment in access/ gender

-Sector capacity building

-Social norm changes

  1. How successfully did AGES reduce barriers to full participation in education or vocational education for highly marginalised girls?
  2. ToC
  • Which factors at household/ school level are affecting the acquisition of literacy, numeracy, and financial literacy skills? How are these factors being affected by socio-economic and political changes in South Central Somalia?
  • Is the project’s ToC adequately reflecting the factors driving learning/ transition/ sustainability?
  • To what extent is the acquisition of social-emotional learning (SEL) competencies affecting learning and transition outcomes? Which subgroups of the population are benefitting the most from SEL competencies?
  • To what extent has participation in the project affected transition pathways for girls enrolled in formal education, ABE, and life skills/NFE programming?
  • What are the key factors driving female youth’s successful transition into wage and self-employment?
  • What are the key factors affecting female youth’s economic empowerment? What additional components would be necessary to maximise success for the subgroups of participants who are lagging behind?
  • To what extent are the skills acquired in the NFE course contributing to economic empowerment? And to the transition into employment?
  1. Inclusiveness
  • Are there subgroups of the targeted population for whom the interventions have been particularly effective in terms of learning and/or transition? If yes, who are they and why have they benefitted the most?
  • Are there subgroups of the targeted population who have not benefitted at all from the intervention, or who have benefitted to a much lesser extent than others, in terms of changes in learning and/or transition? If yes, who are they and why have they not benefitted / benefitted less from the interventions?
  • Are students enrolled in ABE/NFE acquiring literacy and numeracy skills at a similar pace as their peers enrolled in formal education?
  • Are there any differences in learning outcomes between Af-Maay and Af-Mahatiri speakers?
  • To what extent are project interventions effective in addressing barriers to the participation, learning and retention of students with disabilities? Are the interventions particularly effective in addressing barriers for some types of disability, but not others?
  • To what extent are project activities effective in addressing underlying causes of exclusion, including inter-clan / ethnic dynamics and gendered practices limiting girls’ voice and time to study/attend class?
  1. Teaching skills and practices
  • Are there specific literacy / numeracy skill areas that are not improving? Why?
  • Are there ‘plateaus’ in the acquisition of literacy and numeracy skills? To what extent are project interventions effective in addressing those?
  • In a context where education provision is mostly community or privately owned, what works in engaging private sector providers to increase equality and quality?
  • To what extent is positive discipline being applied in classrooms, and what approaches are being adopted by teachers?
  1. Governance
  • To what extent are community education committees addressing factors related to girls’ learning and transition?
  • Have State-specific differences been observed in learning and transition patterns?
  1. Community practices and girls’ empowerment
  • Is the participation of parents/ adolescent girls in economic empowerment activities contributing to increase sustainability of education outcomes, including for other members of the household?
  • To what extent are girls’ roles shifting at the household/ community level as they participate in the project? What are the opportunities and barriers emerging for girls as they take on new roles through the application of academic and leadership skills? To what extent these new roles may be harmful for girls from extremely marginalised groups, such as minority clans and ethnic groups?
  1. Impact of crises and shocks
  • What has been the impact of the ongoing drought and escalation of conflict on household and school conditions? Which subgroups have been particularly affected and why?
  • Are there differences in learning and transition outcomes for girls from the most affected households?
  • How likely are the mid- and long-term effects of these shocks to affect the sustainability of the interventions at school, community, and system levels?
  • To what extent are the project interventions being successful in mitigating the impact of shocks, including for the most vulnerable subgroups of girls?

The following evaluation questions refer only to the NFE cohorts 4 and 5.

  1. Civic action and self-efficacy
  • Is the participation in GEFs contributing to boost girls’ self-efficacy? If yes, to what extent, and for whom?
  • What types of civic action are GEFs involved in? Have those activities contributed to address barriers to adolescent girls and female youth? If yes, how and to what extent?
  • Are project activities contributing to mitigate mental health issues, particularly among the most marginalised girls and female youth? Has the intervention been able to reduce the stigma around mental health issues and increase support-seeking behaviour?
  • To what extent are girls’ empowerment and leadership skills intervention and opportunities perceived as positive by local leaders? Are there specific subgroups whose empowerment is more supported than others?
  1. Access to other development opportunities
  • To what extent has the participation in VSLA contributed to increase female youth participation in income-generation activities? What are the key skills and resources contributing to boost female youth participation in self-employment?
  • To what extent has the inclusive investment in VSLA business plans been successful? What is the profile of the successful/ unsuccessful entrepreneurs? What factors have affected the implementation of business plans (including but not limited to project activities)?
  • Has the intervention contributed to diversify the types of income-generation activities / wage employment female youth are engaged in? What factors have contributed to female youth’s engagement in non-traditional trades/ employment?
  • To what extent are female youth engaging in participatory governance processes through project interventions? Are there differences in female youth engagement in such processes in terms of sector, location, or minority identity?
  1. Access to critical services
  • Has female youth access to critical services such as safety nets and legal services increased over time?
  • To what extent have youth VSLAs and Cluster Level Associations contributed to increase participants’ access to information on critical services? Which subgroups of the targeted population have benefitted the most from such access?
    1. Overall Evaluation Approach

The project uses a longitudinal, mixed-methods pre-post evaluation design with additionality being measured through the use of benchmarks for performance, set for each type of education intervention, corresponding to: an improvement of 0.2 standard deviation over and above the following grade level for formal education; 0.2 standard deviation over and above the corresponding grade for ABE (in relation to the formal grade performance); and minimum achievement levels for literacy and numeracy for at least 70% of the students attending NFE/life skills classes.

    1. Research design

The following methods will be used:

  • Learning assessments (adapted versions of EGRA, EGMA, and a short survey on knowledge of menstrual hygiene management): A sample of 1,420 out-of-school girls (“learning cohort”), randomly selected from the first cohort of girls identified for enrolment in formal education, ABE, and life skills training, was identified at the baseline and will be tracked through the life of the project. This sample will include students from each education ‘track’, or intervention arm – namely formal education (421); ABE (484); and NFE/life skills (515). The sample was drawn from 38 formal schools, 34 ABE centres, and 36 non-formal education classes, randomly selected proportionally to the distribution of implementation locations through the three States, and proportionally to the number of identified girls for each learning track in each State. Girls enrolled in formal education or ABE will be primarily re-contacted at school, while those who have completed NFE/life skills training will be re-contacted at household level. Dropouts who can be located and remain in the same project location will be interviewed at home. The project will provide a database of the 1,420 cohort 1 girls assessed at baseline for re-contacting purposes, including unique identifiers (name, caregiver’s name, school, age and grade at baseline, geographic coordinates of households, caregiver’s phone number where applicable). In addition to the cohort 1 sample, the same learning assessments will be applied with 916 cohort 4 NFE girls assessed at the 2022 baseline for this cohort. The girls will be recontacted in 46 learning centres. Overall, a total of 72 sites will be visited, out of which nine have an overlap between cohorts 1 and 4. Additionally, the baseline for cohort 5 will assess a new sample of 510 randomly selected girls from the same sites where cohort 4 is being implemented.
  • Household survey with parents / caregivers of all girls sampled for learning assessments (estimated as 2,326 households, including 1,420 for the first cohort and 916 for cohort 4/NFE). The household survey includes a head of household module; a caregiver module; and a girl module, to be applied with all girls participating in the learning assessments. The household survey’s caregiver module includes questions related to the household’s socio-economic background; a module on disability, following the Washington Group methodology[8]; questions on education practices; questions on transition and factors influencing it; and questions related to the COVID-19 and exposure to conflict. A modified version of the caregiver module will be applied to older adolescent girls and female youth who are not living with their parents/ caregivers.

The girl module includes questions on her educational experience; disability; leadership skills (Youth Leadership Index questions/ YLI, participation in GEFs and civic action); menstrual hygiene management; safety and security; COVID-19 (preventative measures and impact); exposure to interventions (GEF, savings); addressing mental health issues; engagement in economic empowerment activities and IGAs, and limiting/enabling factors linked to those; participation in governance processes; and access to safety nets. Some questions of the girl module will apply only to older adolescent girls and female youth enrolled in NFE courses. The girl module will be answered by all girls in the sample (i.e. cohorts 1, 4 and 5).

School-level tools

  • school survey will be conducted, including an assessment of school conditions; teacher data; aggregated school enrolment; and head counts in grades 1-5 (and NFE cohort 5). This survey will be applied in all sample sites with a formal school participating in the project.
  • Classroom observations using structured timed tools (applied to teachers in Grades 1-5 in sampled schools, and with NFE facilitators)
  • Qualitative data collected from:
    • Parents (focus groups, separately with mothers and fathers);
    • Girls (vignettes and risk mapping);
    • Teachers (focus groups);
    • CECs (focus groups);
    • Religious leaders / authorities (in-depth interviews).

Qualitative data will be collected from 12 sample sites. All qualitative data will be recorded, integrally transcribed, and translated.

Quantitative data will be collected using electronic tools.

All tools will be piloted prior to data collection and refined as necessary (with exception of the learning assessments, which will be piloted and calibrated in advance by CARE). The external evaluator will be responsible for piloting the tools, while CARE will liaise with the Fund Manager and the MOECHE for approval.

Potential adaptations in response to COVID-19 and security restrictions

Current conditions allow for in-person data collection in South Somalia, provided enumeration teams adhere strictly to safety protocols for their own safety and the safety of project participants and other stakeholders at community level. CARE does recognise, however, the potential for rapid shifts in field conditions and the need to plan for alternative scenarios in case of a dramatic escalation in infection rates and/or movement restrictions due to other reasons, including instability, conflict, and natural disasters. In case those scenarios occur, CARE will work with the external evaluation and data collection company / companies and the Fund Manager to agree on adaptations, including phone-based data collection with abridged tools and modified versions of learning assessments.

    1. Sampling for the fifth cohort’s baseline

The sampling process is described above and in the project’s MEL Framework. To allow for a more streamlined procedure, increase comparability of results, and avoid extraordinarily high evaluation costs, the cohort 5 baseline will be conducted in the same sites (i.e. learning centres) as cohort 4. Those sites were selected in 2021 using stratified random sampling (proportional to the distribution of the cohort 4 population). The project will provide a list of all enrolled girls in those learning centres (sampling framework) to the external evaluator, who will be required to help finalise and operationalise the sampling approach for both qualitative and quantitative samples. In each sample site, students will be randomly selected from the list of enrolled out-of-school girls. Please refer to section 6.5 above for the required sample sizes. The final sample should be representative of the fifth NFE cohort and collected in an appropriate manner, to ensure:

  • Reasonable ability to generalise the intervention’s effectiveness to similar contexts; and
  • Reasonable ability to generalise the insights into what works and why for similar contexts.
    1. Ethical protocols

The evaluation must consider the safety of participants and especially children and vulnerable individuals at all stages. The selected evaluation company will need to demonstrate how they have considered the protection of children through the different data collection stages, including recruitment and training of research staff, data collection, data analysis, and report writing.

The selected evaluation company is required to set out their approach to ensuring complete compliance with international good practice with regards to research ethics and protocols, and in particular to safeguarding children, vulnerable groups (including people with disabilities) and those in fragile and conflict-affected states. Consideration should be given to:

  • Administrative, technical and physical safeguards to protect the confidentiality of those participating in research;
  • Physical safeguards for those conducting research;
  • Data protection and secure maintenance procedures for personal information;

Parental consent concerning data collection from children or collation of data about children;

  • Age- and ability-appropriate assent processes based on reasonable assumptions about comprehension for the ages of children and the disabilities they intend to involve in the research;
  • Age-appropriate participation of children, including in the development of data collection tools;
  • Immediate communication to CARE, following appropriate channels as determined by CARE’s policies, of any occurrences or reports of child protection, abuse or harassment cases. All enumerators will receive a briefing on CARE’s policies on child protection and prevention of sexual harassment and abuse (PSHEA).
    1. Risk and risk management[9]

Risk management plan: It is important that the successful bidder has taken all reasonable measures to mitigate any potential risk to the delivery of the required outputs for this study. Therefore, the selected evaluation company should submit a comprehensive risk management plan covering:

  • The assumptions underpinning the successful completion of the proposals submitted and the anticipated challenges that might be faced;
  • Estimates of the level of risk for each risk identified;
  • Proposed contingency plans that the bidder will put in place to mitigate against any occurrence of each of the identified risk;
  • Specific child protection risks and mitigating strategies, including reference to the child protection policy and procedures that will be in place; and
  • Health and safety issues that may require significant duty of care precautions.
    1. Data quality assurance

Quality assurance plan: the selected evaluation company is required to submit a quality assurance plan that sets out the systems and processes for quality assuring the research process and deliverables from start to finish of the project. This plan should include the proposed approaches to:

  • Piloting of all tools;
  • Training of enumerators and researchers conducting the mixed-methods primary research, including in research ethics;
  • Logistical and management planning;
  • Field work protocols and data verification including back-checking and quality control by supervisors;
  • Phone-based data collection quality assurance (if applicable);
  • Data cleaning and editing before any analysis;
  • Database collation; and
  • Qualitative data checks.
    1. Existing Information Sources

In the first instance, the selected evaluation company should refer to the GEC website:

https://www.gov.uk/guidance/girls-education-challenge for general information concerning the Girls’ Education Challenge.

The selected evaluation company should refer to the following GEC programme documentation:

  • Grant Recipient Handbook
  • Evaluation Guidance
  • Logframe and workplan guidance

The selected evaluation company should refer to the following GEC project documentation that includes:

  • Project logframe;
  • Project MEL framework;
  • Project documents related to the COVID-19 adaptations.

The selected evaluation company should also refer to relevant country data and information that is currently available, as required, to prepare the proposal.

  1. SCOPE OF WORK

4.1. Professional Skills and Qualifications

Qualifications: the selected evaluation company is required to clearly identify and provide CVs for all those proposed in the Evaluation Team, clearly stating their roles and responsibilities for this study. Bidders are required to provide CVs by component, i.e. (1) data collection and (2) evaluation design, analysis, and reporting. In case the same team is proposed for both components, bidders should indicate that this is the case.

The proposed evaluation team should include the technical expertise and practical experience required to deliver the scope of work, with regards to:

Component 1: Data collection

  • Data collection planning: The team should include skills and expertise required to design, plan and conduct large scale quantitative and qualitative data collection in fragile contexts;
  • Relevant subject matter knowledge and experience: Knowledge and experience required on conducting research with children, the education sector, disability and gender to ensure that the research methods and approaches used in the field are as relevant and meaningful as possible given the aims and objectives of the project and the context in which it is being delivered;
  • Data collection management: Manage a large-scale and complex research process from end-to-end, including remote management of teams;
  • Primary research: Gender-sensitive implementation of primary quantitative and qualitative research in potentially challenging project environments, such as fragile and conflict affected states – this could include the design of longitudinal household panel surveys, advanced versions of EGRA /EGMA tests, in-depth interviews, focus groups, participatory qualitative exercises with children, etc.;
  • Country experience: It is particularly important that the team has the appropriate country knowledge /experience required to conduct the research;
  • Ability to operate in project regions: The bidder should be able to mobilise experienced enumerators able to collect data in targeted areas and knowledgeable about local dynamics;
  • Experience in electronic data collection and phone-based, virtual data collection;
  • Quality assurance, data management and data cleaning: Ability to supervise the collection, entry (if required), cleaning and management of large datasets. This includes backtranslation of surveys; field supervision of teams and remote supervision/ checks whenever applicable; systematic daily checks of incoming electronic data during data collection; database management; and transcription, translation and verification of qualitative datasets; and
  • Safety considerations: Ensuring the whole study process adheres to best practice for research with children, including the implementation of CARE’s child protection policy and procedures to ensure safety of participants. Note that the selected bidder is expected to be able to show that they have a child protection policy in place to safeguard children that the research team would be in direct contact with through the research activities.

Component 2: Evaluation design, data analysis and reporting

  • Study design: The team should include skills and expertise required to design, plan and conduct rigorous, mixed-methods impact evaluations in fragile contexts;
  • Skills in quantitative and qualitative data collection and analysis, drawing findings from multiple sources and handling potential contradictions between datasets;
  • Relevant subject matter knowledge and experience: Knowledge and experience required on conducting research with children and adolescents, the education sector, disability and gender to ensure that the evaluation design and research methods are as relevant and meaningful as possible given the aims and objectives of the project and the context in which it is being delivered;
  • Evaluation management: Manage a large-scale and complex research process from end-to-end;
  • Primary research: Gender-sensitive design, management and implementation of primary quantitative and qualitative research in potentially challenging project environments, such as fragile and conflict-affected states – this could include the design of longitudinal household panel surveys, adapted versions of EGRA /EGMA tests, in-depth interviews, focus groups, participatory qualitative exercises with children/ adolescents, etc.;
  • Country experience: It is particularly important that the team has the appropriate country knowledge /experience and ability to interpret findings from a contextual perspective, as required to conduct the research;
  • Information management: design and manage sex- and disability-disaggregated data and information systems capable of handling large datasets for MEL purposes, including cross-referencing different datasets;
  • Statistical analysis: a range of statistical modelling and analysis of impact data; highly proficient user of SPSS or STATA; and qualitative data analysis techniques, preferably considering emerging codes;
  • Data management and data cleaning: Ability to supervise the collection, entry (if required), cleaning and management of large datasets. In case the two Components are awarded to different bidders, the company selected for Component 2 will provide technical guidelines to the company contracted to deliver Component 1 on data management and cleaning.
  • VfM assessment of education projects: Education economics expertise to conduct cost-benefit and cost-effectiveness analyses as part of the assessment of the project’s VfM; and
  • Safety considerations: Ensuring the whole study adheres to best practice for research with children including the implementation of child protection policy and procedures to ensure safety of participants. Note that all bidders are expected to be able to show that they have a child protection policy in place; although Component 2 does not include direct contact with children, a child protection policy is essential to ensure an appropriate approach to data analysis and reporting.

The day–to–day project management of this study will be under the responsibility of Paul Odhiambo, M&E and Knowledge Manager, AGES project/ CARE Somalia.

4.2. Deliverables and Schedule

Expected Tasks

Component 1: Data collection

  1. Develop a detailed work plan for data collection, data entry (where relevant – it is anticipated that most of the data will be collected electronically), data cleaning, transcription, and translation.
  2. Translate data collection tools and training protocols from English to Somali. Note that back-translation will be required for accuracy. The selected bidder is also expected to be able to use translations in Af-Maay where applicable, allowing students to respond in their mother tongue to the girl survey.
  3. Upload all relevant data collection tools into an electronic format.
  4. Lead the training of enumerators and team leaders, technically supported by the evaluation team (Component 2) and CARE. The data collection company will follow the guidelines and the training plan created by the evaluation team leading on Component 2; and will implement quality control protocols and guidelines for enumerators previously reviewed and approved by the external evaluation team and CARE’s M&E team.
  5. Prepare all training and data collection tools and related supplies for enumerators, including mobile phones/ tablets; printouts of assessments and reference materials.
  6. Conduct all data collection in the field for each tool described above, including testing of tools and actual data collection.
  7. Support in data collection related logistics, including but not limited to deployment of enumerators and supervisors to the target areas; and where required coordination with CARE to access each area with approval from local authorities.
  8. Upload datasets completed by the team on a daily basis for verification.
  9. Create electronic data entry forms (databases) as necessary in case of data entry.
  10. Qualitative data will be completely transcribed in Somali and translated into English. Spot-checking back-translation will be required for qualitative data.
  11. Conduct data entry, where necessary, simultaneous to data collection to prevent data delays.
  12. Conduct all necessary data quality and ethical control measures.
  13. Provide a complete set of physical and electronic data collection documents, filed, and organized as per guidelines provided by the External Evaluation Company, upon completion of data collection and entry. This includes but is not limited to filled surveys, questionnaires, FGD/ interview notes, daily survey logs, voice files (for electronic qualitative data collection), pictures, etc. Please note that CARE’s guidelines do not authorize pictures of respondents due to the need for confidentiality.

Component 2: Evaluation design, data analysis and reporting

1. Review the project’s MEL framework, other relevant project documents and external literature related to the Somali context and in particular, the education sector in country;

2. Submit an inception report (draft and final), that outlines the research methodology, including data collection tools, detailed analysis framework and detailed work plan outlining all tasks to be completed by each of the members of the study team;

3. Finalization of tools, including adaptations post-piloting and back-translation supervision;

4. Technical support for enumerator training content and toolkits (qualitative and quantitative) and supervise/monitor training on data collection tools and their application;

5. Follow up on and directly supervise enumerators’ work in the field during data collection (qualitative and quantitative) and provide the necessary technical support to the data collection team to ensure quality of data, including overnight checks of electronic datasets;

6. Validate all datasets, working in coordination with the data collection company, and collate data as necessary for analysis;

7. Analyse the collected data in accordance with the agreed analysis framework.

8. Prepare a comprehensive draft and final study report, according to the format provided by the Fund Manager, and inclusive of the GEC Outcomes Spreadsheet if required.

Deliverables

In reference to the scope of work above, the consultant team is expected to accomplish and submit the following:

Component 1: Data collection

  1. Workplan outlining all tasks to be completed within the duration mentioned for this assignment, responsible persons, timeframe for completion of each task and resources required for each task (within five days of signing the contract with CARE).
  2. Detailed data collection plan, as part of inception report, in line with the approved M&E framework and sampling framework provided by CARE. It should be noted that data collection dates need to be accurate as these will be shared with the GEC Fund Manager (PwC).
  3. Electronic versions of surveys, including translations of survey instruments from English into Somali, to be verified through back-translation (conducted by a second independent translator contracted by the data collection company) and secondary spot-checks of translated content vis a vis the original conducted by CARE.
  4. Complete list of enumerators/supervisors per location, including contact details.
  5. Daily uploading of data and weekly data collection report.
  6. Daily incident reports as necessary, including but not limited to any child protection or PSHEA cases identified.
  7. Complete set of files, hard copies of surveys, audio/ image/ video files and daily data collection reports/ data collection logs signed by enumerators and field supervisors.
  8. Database for data entry (only in case paper-based surveys are used for any tool), to be reviewed and approved by the evaluation company (Component 2). Data entry (if required) should start simultaneously with data collection and data should be uploaded on a daily basis.
  9. Complete transcriptions of qualitative data, in Somali and in English.
  10. Clean datasets, to be reviewed and approved by the evaluation company/ Component 2 (which will assess and certify data quality). 100% of the planned data should be included and the final version of the database should not include any typos, out of range responses for any variable and/or illogical responses. The data collection company will respond promptly to requests by the evaluation company for correction or verification of problems in the datasets, both during data collection and after completion of the full datasets, until the datasets are considered to be final.
  11. Final report, to include discussion of data collection challenges and limitations, including a summary of data collection activities and the timeline of data collection, number of total surveys collected and number of refusals to participate for each location, as well as detailed data collection log per location as an annex.

Component 2: Evaluation design, data analysis and reporting

  1. An inception report including:
  • Research methodology including the sampling methodology for the baseline of the cohort 5 and the analysis framework for the study;
  • Draft data collection tools;
  • Detailed work plan outlining all tasks to be completed by each of the members of the consultant team for the duration of the study.
  1. Data collection protocols;
  2. Technical support for training guidelines for enumerators incl. training tool kits, and training session plans;
  3. In depth quantitative and qualitative analysis of data;
  4. Complete clean datasets in SPSS, inclusive of complete codebooks, and where applicable, consolidated datasets (household survey + learning assessments);
  5. Completed cohort tracking tool for the sample, including unique identifiers for all girls;
  6. Complete transcriptions and translations of qualitative data, as well as sound files;
  7. SPSS syntax and output files reflecting the analysis conducted;
  8. A 150-300 page draft report (in MS Word), following the format required by the Fund Manager.
  9. Presentations of preliminary findings and recommendations validation and feedback with CARE and other stakeholders, including the project’s Steering Committee and donors.
  10. Final report with corresponding finalized annexes in English.
  11. Summary report with key findings, using a format and language accessible to non-technical audiences.

Project milestones: bidders are required to include in their detailed work plans the milestones set out below.

Component 1: Data collection

Milestone

Timeframe/ number of days

Training outline and materials

Draft Inception Report submitted for review and comments

Review of inception report completed and comments returned to supplier/consultant

Final Inception Report submitted

Electronic versions of all tools, inclusive of translation

Training

Tool piloting

Data collection starts

Weekly progress reports

Data collection completed

Draft datasets submitted

Clean datasets submitted

Data collection report submitted

Component 2: Evaluation design, data analysis and reporting

Milestone

Timeframe/number of days

Literature/document review / review of project’s theory of change, impact logic

Draft Inception Report submitted for review and comments

Presentation to Evaluation Steering Group

Review of inception report completed and comments returned to supplier/consultant

Final Inception Report submitted

Training

Tool piloting

Study starts

Study completed

Data analysis and drafting of the report

Draft Study Report submitted for review

Presentation to Evaluation Steering Group

Review by Project Management and stakeholders completed / comments provided to Supplier/Consultant

Supplier/Consultant addresses comments and revises the study

Final report submitted

The data collection and independent evaluation company/ companies will be expected to identify a Project Director and Project Manager for communication and reporting purposes. At the inception meeting, the Project Manager (s) will be expected to submit a full contact list of all those involved in the second midline evaluation.

The selected bidder (s) will be expected to report to the Evaluation Steering Group and attend all meetings as agreed with the Project Evaluation Manager. The successful bidder (s) will be required to submit to the Project Evaluation Manager weekly progress reports (by email) during the study periods summarising activities /tasks completed to date (percent achieved), time spent etc.

In addition to the Evaluation Steering Group, the Ministries of Education will also be part of the study. Regional/State MoE staff will liaise with the sampled schools to ensure their support to the study, working closely with CARE and Consortium Partners. No MOECHE/ MOE staff will participate in the actual data collection exercise to ensure an independent process. The MOECHE research coordinator and gender focal person will provide input to the draft evaluation tools and report.

  1. SUBMISSION

The deadline for submission of bids is 17th December 2022. Interested bidders are welcome to request for the Project Logframe (Annex 1) or submit questions until 13th December 2022. To Paul.Odhiambo@care.org.

Annex 1: Project Logframe

Annex 2: Risk Management Template

Risk type

Risk

Likelihood

(1-5, 1 – not likely, 5 – highly likely)

Severity

(minor; moderate; major or severe)

Mitigation

(please briefly outline the measures you propose to mitigate these risks)

How to Apply.

All applications MUST be accompanied by a technical and financial proposal.

Interested consultants or firms are expected to submit their applications to: SOM.Consultant@care.org .Please **indicate “**Second Midline Evaluation Consultancy Services for AGES project” as the subject heading not later than 15, December 2022.

Female applicants with requisite Somalia experience are highly encouraged.

CARE is an equal opportunity employer promoting gender, equity, and diversity. Female candidates are strongly encouraged to apply. Our selection process reflects our commitment to the protection of children from abuse

How to apply

All applications MUST be accompanied by a technical and financial proposal.

Interested consultants or firms are expected to submit their applications to: SOM.Consultant@care.org .Please **indicate “**Second Midline Evaluation Consultancy Services for AGES project” as the subject heading not later than 15, December 2022.

Female applicants with requisite Somalia experience are highly encouraged.

CARE is an equal opportunity employer promoting gender, equity, and diversity. Female candidates are strongly encouraged to apply. Our selection process reflects our commitment to the protection of children from abuse

This job has expired.