Project End-line Evaluation

Project Name: Inspiring Somaliland and Somalia Youth through skills training and creation of employment opportunities

Countries: Somalia and Somaliland

Project Background

Africa Educational Trust(AET) has been operating in Somalia since 1996 and is currently implementing programs in Somalia and Somaliland. AET works closely with Ministries of Education, local and international non-governmental organizations across the Education sector. With large youth empowerment and technical vocational education and training program portfolio in Somalia and Somaliland, AET works to provide life-changing services to youth and young adults in the two countries through a combination of skills training, mentorship and microfinance so that they can successfully establish themselves in business

Recognizing the lack of Technical Vocational Education and Training (TVET) opportunities for youth, and the fact that limited options for education and employment make young people highly vulnerable to recruitment by extremist groups and criminal gangs, AET was awarded 400,000 euros by Enable to implement “**Inspiring Somaliland and Somalia Youth through skills training and creation of employment opportunities**”. The Youth Employability and Participation project is a two-year project, beginning December 2019, and ending on 28^th February 2022, following a two-month no-cost extension. Target beneficiaries of the project are 400 young people aged 15 to 35 years who are outside of education, training, and employment. Priority will be given to young people from poor and vulnerable households where educational achievements are lower, where poverty spans across generations, and who receive no support from international remittances. Of these, at least 250 will be among the most disadvantaged, including young people with disabilities and IDPs displaced by conflict or drought to urban areas. The project is being delivered directly by AET working closely with relevant local state ministries and Microdahab(the development arm of Dhabshill bank)

The broad objective of the project is to contribute towards poverty reduction and SDGs 1, 4, and 8 by demonstrating an effective model for reducing youth unemployment and vulnerability in Somalia. The specific objective is to enable young people in Somalia and Somaliland to have increased opportunities for employment and entrepreneurship

The project is working towards the following outcomes:

1. Increased access for disadvantaged youth to vocational skills training, relevant to market needs.

For young people who have missed out on education, foundational literacy and numeracy skills are an essential first step in accessing vocational training opportunities as well as in equipping them with the skills to keep business records in the future. A market assessment will identify potential opportunities for work/self-employment and matching vocational education and training (VET) courses. Appropriate guidance and mentorship will then be targeted to help youth select vocational skills training that is relevant to the labor market needs.

2. Enhanced transition from vocational training to work

The project will work with the government and the private sector to facilitate links between VET and the labor market that can promote meaningful employment opportunities. The relevance of vocational training will be enhanced through work placements with existing businesses and the integration of entrepreneurship and financial literacy training. Finally, the project will facilitate access to microfinance through our partnership with MicroDahab, providing a means for youth to apply their skills in establishing business initiatives and stimulating micro-economic development in their communities.

3. Integration of entrepreneurship and financial literacy training in VET courses.

The Ministries of Education aim to make entrepreneurial training a compulsory component of VET provision. The project will support this strategy by working with the MoEs and VET providers to design and deliver the entrepreneurship and financial literacy module and embed it into existing Vocational Education Training (VET) programs. This will ensure future VET graduates acquire the necessary entrepreneurial skills to help them transit to the labor market.

The expected results (outputs) of this project are:

• A labor market analysis report is produced to identify market opportunities and skills needed to access these
• Young people pass literacy and numeracy skills training and are able to access vocational training (target – 400 youth)
• Young people receive mentorship and are able to make informed choices of vocational skills training relevant to market needs.
• Young people complete vocational skills training alongside entrepreneurship and financial literacy training (target – 400 youth).
• Young people are able to gain employment or access micro-finance to establish themselves in business within the project lifetime (target – at least 250 youth).
• Guidelines for the delivery of entrepreneurship and financial literacy training and how this can be integrated into VET courses are developed and shared with the MoEs and VET providers.

Purpose of the end line evaluation

The purpose of this evaluation is to assess the performance of the project and capture project achievements, challenges, and best practices to inform future similar programming. It will also review the recommendations of the first and second donor-led monitoring missions and assess the extent to which these were implemented. The evaluation will also ensure accountability towards Enable as a donor and the beneficiaries of the project and capture learning from the project implementation for all stakeholders. The evaluation will therefore, identify key lessons learned, challenges, and the flexibility of the project to adapt and respond to changes during the life of the project and sustainability of Technical Vocational Education and Training in Somalia and Somaliland

The end of project evaluation will have three objectives:

Objective 1: Evaluate to what extent the “**Inspiring Somaliland and Somalia Youth through skills training and creation of employment opportunities**” has delivered effective, efficient, relevant, and timely activities to beneficiaries as set out in the project log frame.

Objective 2: Assess whether the collaboration between AET, Ministries of Education and Science (Non-formal Departments), and Microdahab has added value to the interventions with a positive effect on beneficiaries and other stakeholders. What has contributed to this added value and what has not?

Objective 3: Identify and assess key lessons learned, challenges and draw recommendations for future programming of Technical Vocational Education and Training in Somalia and Somaliland

Objective 4: To establish feedback and recommendations from employers and providers of work placements.

Objective 5: To conduct a Tracer Study of the TVET graduates.

This will be a complementary study to end-line evaluation. The objective of the study is to assess the employment destination of the trained youth and measure the extent to which skills acquired through the program has improved TVET graduates living status

Objective 1: To what extent the “**Inspiring Somaliland and Somalia Youth through skills training and creation of employment opportunities**” has delivered effective, efficient, relevant, and timely activities to beneficiaries as set out in the project log frame.

I. Effectiveness:

The end line evaluation should assess the following:

• To what extent have the planned objectives in the log frame of the project been reached?
• To what extent have the project activities contributed to the overall goal? Was the project effective in increasing the skills of TVET graduates to improve their living status and respond to humanitarian needs?
• What were the major factors influencing the achievement of the objectives of the project?
• Identify any un-planned outcomes of the project and their impact on beneficiaries.
• What opportunities for collaboration have been utilized and how have these contributed to increased effectiveness? or otherwise?
• Have proper accountability and risk management framework(s) been in place to minimize risks on project implementation

II. Efficiency

• How efficient was the delivery of the project by AET, not only in terms of expenditure but also in terms of implementation of activities?
• Was the project activity implementation (modality), considered to have been cost-efficient, while not compromising quality?
• What would have been opportunities within the project to reach more beneficiaries with the available budget or to reduce costs while reaching at least the same number of beneficiaries without compromising quality?
• Were alterations made to the project design in terms of collaboration during the implementation phase based on the reality on the ground?
• What were the outcomes of these choices for effective and efficient project implementation?

III. Relevance /Impact

• How relevant were the objectives and activities implemented by the project in addressing the need of youth in Somalia and Somaliland to generate a livelihood?
• How do beneficiaries perceive the relevance of the project and how have the activities that were implemented improved their lives? Are there any stories of change?
• How has the collaboration between AET, Ministries of Education and Science (Non-formal Departments), and Microdahab contributed to appropriate response of specific needs and priorities of the beneficiaries?
• To what extent was the project able to adapt and provide an appropriate response to context changes and emerging local needs, and the priorities of beneficiaries?
• Was the project design and interventions timely in responding to the needs on the ground?
• Were the activities implemented in a timely way in comparison to project planning?
• Were funds available in time during the implementation of the activities?
• To what extent did the collaboration between AET, Ministries of Education and Science (Non-formal Departments), and Microdahab contribute to efficient and timely coordination of logistic activities and processes?

IV. Quality:

The end-line evaluation should assess the overall quality of the implementation. It is important to include beneficiaries’ opinions on the quality of the services received.

• What mechanisms are in place to track project implementation of the AET projects? (i.e. internal monitoring, evaluation, accountability, learning (MEAL) and quality assurance mechanisms)?
• How have they been utilized to increase quality within the project?
• Did the quality of activities delivered by the project meet the needs and expectations of the beneficiaries? What do beneficiaries feel could be improved in contribution to improving local capacity?
• To what extent did the project interventions contribute to building long-term community capacity?
• To what extent was the project participatory in all the project cycle?

Objective 2: A**ssess whether the collaboration between** AET, Ministries of Education and Science (Non-formal Departments), and Microdahab has added value to the interventions with a positive effect on beneficiaries and other stakeholders. What has contributed to this value addition, and what has not?

Learning:

• Which of the interventions, approaches, and modalities/strategies have been most effective according to AET and Ministries of Education and Science (Non-formal Departments)?
• Is there any substantial evidence on how project learning was generated and applied to improve the delivery or effectiveness or efficiency of activities?
• Who benefited from shared learning experiences (e.g. quarterly, joint field visits, workshops provision on best approaches and methodology)?
• How did the different actors learn from these experiences?

Complementarity and harmonization:

• Are there any concrete examples of successful models of collaboration of AET with other partners on a geographic level, not just in terms of avoiding duplication but increasing complementarity and integrated programs affecting the reach and impact on beneficiaries?
• What are barriers and/or enablers to this?
• To what extent were the activities of the project complementary to the work of other stakeholders, i.e. prevented duplication and contributed to the larger response activities in Somalia and Somaliland?

Visibility:

• What measures have been taken to create visibility of the project’s added value towards line ministries, the general public (including beneficiaries), Governments, and other INGO forums?

Objective 3: Identify and assess key lessons learned, challenges and draw recommendations for future programming of Technical Vocational Education and Training in Somalia and Somaliland

The end line evaluation should at least include one lesson learned and recommendation per evaluation category, i.e. effectiveness, efficiency, relevance, etc.

What are the key lessons learned per objective? To what extent has the delivery of response activities contributed to effective, efficient, relevant, and timely delivery of aid and enhanced impact for the beneficiaries?

Sustainability

• Will the changes caused by this project continue beyond the life of the project?**
• What mechanisms has AET put in place to sustain the key project outputs and outcomes?**
• How has the project worked with local partners to increase their capacity in a sustainable way?**
• What motivations /mechanisms exist for partners to continue playing these roles?**
• What are the risks facing the sustainability of project outputs and outcomes?**

Objective 4: To establish feedback and recommendations from employers and providers of work placements.

• How many youth undertook work placements/work experience including demographic information of gender, age, socio-economic situation (IDP, refugee) or if having a disability?
• What was the experience of employers/work placement providers? Have they provided work placement/apprenticeships before? What did they find challenging? What did they perceive as the benefit to them/their business?
• If they provided for a student with disabilities, what additional support did they receive? What would they recommend for support to make the work placement effective?
• What recommendations would workplace providers make for future projects?

Objective 5: To conduct Tracer Study with the TVET graduates.

The tracer study should assess the following

• To what extent have the skills gained by TVET graduates been or not been utilized? what are the main reasons for not utilizing the skills
• Are there any concrete examples to show that skills gained improved graduate economic status?
• How was the training delivered? Are graduates satisfied with the quality of training?
• How long does it take the trained youth to find employment?

1. Methodology

The methodology will be developed by the consultant, as well as all relevant tools and presented in the inception report. The first three strategic objectives mentioned above should be assessed, including all research questions under each objective.

The data collection should include the use of a number of approaches to gain a deeper understanding of the outcomes of the project, including:

• Desk review of background documents (project document, project monitoring data, progress report, mid-term review report, field visit reports etc).
• End line evaluation survey with TVET graduates/beneficiaries in project implementation sites
• Key informant interviews (e.g. with AET project staff members, i.e. project manager, Project officer, and key community members/beneficiaries, Employers, Microdahab, and representatives from the line ministries) to gather substantial anecdotal evidence on the effectiveness, efficiency, relevance, and timeliness** of the project activities implementation and delivery.
• Focus group discussions (e.g. with stakeholders, community members, TVET undergraduates, and CEC representatives, The FGD will serve as input for the narrative anecdotal evidence.

Next to the data collection method, an appropriate and strategic sampling method should be selected e.g. snowball sampling, purposeful random sampling, or mixed purposeful sampling methods.

2. Deliverables

The consultant is expected to lead, accomplish and submit the following deliverables within the agreed timeframe and budget:

An inception report, which will serve as an agreement between parties on how the evaluation will be conducted. Items to address:

• Understanding of the issues and questions raised in the TOR data sources
• How to assess the questions in the ToR
• Research methodology, including suggested sample and size, schedule of activities, and traveling(timeline)
• Appropriate validated draft data collection tools (e.g. methodological guidelines, group interview questions)
• A maximum of 40-page draft and final evaluation report (in MS Office and PDF for final), excluding annexes and in English
• A maximum of 20-page draft and final tracer report (in MS Office and PDF for final), in English

Both reports should be in the format indicated below, to be submitted to the Executive Director of AET. It is preferable to illustrate the results by appropriate graphs, visuals, tables, and/or a dashboard with an accompanying explanatory text. The report should consist of:

a) Executive Summary in bullets (max. 2 pages)

b) Introduction

c) Methodology, including sampling and limitations

d) Analysis and findings of the evaluation. The analysis should be done according to the objectives 1- 5 including an implementation strategy for the recommendations

e) Address concerns, lessons learned and comments from AET

f) Stories of change and quotes from respondents

g) Conclusions for each of the end line evaluation objectives

h) Recommendations for future projects

i) Annexes:

• Relevant maps and photographs of the evaluation areas where necessary
• Bibliography of consulted secondary sources
• Finalized data collection tools
• List of interviewees with accompanying informed consent forms
• PowerPoint presentation of preliminary findings to AET

3. Indicative timelines

The data collection phase in the field is to be confirmed between the consultant and AET, but ideally would start in the second week of February 2022, with the final report deadline to AET by the first week of March 2022

Phase 1: Inception phase

Deliverables: Draft inception report

Working days indication: 1

Phase 2: Tools development phase

Deliverables: Deliverable 1: final inception report including a budget, methodology, and qualitative research tools, approved by the Executive Director of AET

Payments: 30% of total budget

Working days indication: 2

Phase 3: Data collection phase

Deliverables: Desk review, interviews in, and field visits to Somalia and Somaliland

Working days indication: 10

Phase 4: Evaluation report phase

Deliverables: Draft Evaluation Report for comments by AET team

Working days indication: 3

Deliverables: Deliverable 2: Final Evaluation Report

Payment: 70% of total budget

Working days indication: 2

4. Qualifications and experience required

• At least Master’s degree in Education, International Development Studies, Humanitarian Leadership Program, or a related field
• At least 7 years’ experience in working with humanitarian sectoral programs relating to education (Technical Vocational Training) and livelihood program
• Demonstrated experience with quantitative and qualitative research, database management, and statistical data analysis
• Experience of working in Somalia
• Experience in evaluating TVET programs
• Proven record of communicating with beneficiaries.
• Understanding of TVET policy, standards, and protocols
• Ability to assess and further develop a conceptual evaluation tool
• Relevant subject matter knowledge and experience regarding the thematic areas of this TVET project
• Ability to deal with hardship and remote area fieldwork

Preferred

• Strong understanding of humanitarian and evaluation ethics and a commitment to ethical working practices
• Experience of working in insecure environments in Somalia and managing security risks
• Action-oriented and evidence-based approach and a strong drive for results
• Highly developed self-management, and communication skills

5. Guiding Principles and Values

Adherence to AET Code of conduct, Child Safeguarding practices, and confidentiality when interviewing or photographing children. Gender mainstreaming is key to AET and the donor; therefore, the lead consultant will have to ensure that the research team is gender-balanced, ensuring that females are available to interact with female beneficiaries and vice versa. The consultant will also take into account principles of impartiality, independence, objectivity, participation, collaboration, transparency, reliability, privacy, and utility throughout the process.

6. Selection process

AET will use its internal guidance, checklists, and an interview process to select the successful consultant. The guidelines require the consultant to submit a proposal explaining their comprehension of the ToR, and how they would approach this assignment, with a summary of their methodology, especially in terms of how they plan to meet the objectives, including a period and budget. This should include a team composition with a lead consultant and at least one other experienced evaluator and a CV of each person to be involved in the assignment, including relevant experience, a detailed budget, and time availability.