Final Impact Assessment Improved eye health in Kenya through Peek technology/Vision Impact Project. Full Time

Terms of Reference for Evaluation

Evaluation Summary

Project: Improved eye health in Kenya through Peek technology/Vision Impact Project

(Project Numbers: BMZ-P5248; CBM –P10015 & P10023(Bomet) and P10020&P10025 (Sabatia)

Name of Partner Organisations: Sabatia Eye Hospital and AGC Tenwek Hospital

Project start and end dates: 01-11-2021 to 31-03-2025

Evaluation Purpose: To comprehensively assess the project period examining the project’s achievement of its primary goal of strengthening of eye health systems to generate learning for future programming, policy development, and scaling of successful interventions.

Evaluation Type: Final Impact Assessment

Contracting organisation: CBM Christoffel-Blindenmission Christian Blind Mission e.V

Contact person: David Munyendo

Primary Methodology: Mixed Methods as appropriate

Proposed Start date: 25th April 2025

End date: 30th June 2025

Anticipated Evaluation Release Date: 30th June 2025

i. Project Description

About CBM

CBM Christo­ffel-Blindenmission Christian Blind Mission e.V is an international Christian development organization, committed to improving the quality of life of people with disabilities in the world. CBM addresses disability as a cause and consequence of poverty and works in partnership with implementing partners and government to create an inclusive society for all.

About Vision Impact Project

CBM supports community and school eye health programmes in Kenya to ensure that no one goes needlessly blind due to avoidable blindness. The Vision Impact Project (VIP) has been implemented in partnership with AGC Tenwek Hospital (Bomet County) and Sabatia Eye Hospital (Vihiga County) and co-funded by the German Federal Ministry for Economic Cooperation and Development (BMZ) for the period November 2021 to March 2025. The overall goal of VIP was to reduce prevalence of visual impairment and avoidable blindness in Vihiga and Bomet counties in Kenya through strengthening of eye health systems across primary, secondary, and tertiary levels using Peek technology. VIP targeted to screen and treat at least 60% of the general community population and at least 80% of children in school from 1,583 primary/secondary schools in the 2 target Counties. To achieve this, the project prioritized investment in 5 result areas namely:

• Eye health data in Vihiga and Bomet Counties is improved.
• Ophthalmologic facilities of Vihiga and Bomet Counties are functional and able to apply Peek technology.
• Ophthalmic care in the communities of Vihiga and Bomet Counties is increased.
• Ophthalmic care in primary and secondary schools in Vihiga and Bomet Counties is increased.
• Eye health governance is strengthened at the county level.

Feasibility Study and Mid-Term Review

A feasibility study was conducted at the beginning of the project and key gaps were noted in areas including but not limited to inadequate infrastructure and equipment, low demand for eye services, inadequate eye health work force, weak eye health governance in the county and inadequate financing in the county.

In 2024, CBM’s Inclusive Eye Health department conducted a mid-term review of the project which highlighted great progress towards strengthening eye health systems in the target counties.

Detailed reports shall be availed to the successful candidate.

ii. Evaluation Objective and Intended Use

The final evaluation shall provide evidence of the achievements of the project following an analysis of the OECD DAC criteria. As an innovative approach has been used, i.e. Peek technology for eye screenings, the focus shall lie in the assessment of its effectiveness, efficiency and equity. The stakeholders want to understand how the use of Peek technology was able to enhance overall eye health services in the target Counties. It is also important to assess how far ophthalmic care in the communities and schools was improved and how the County level health governance was strengthened.

iii. Scope of the evaluation

The evaluation will cover the full period of implementation from November 2021 to March 2025 and will cover areas including but not limited to project design, planning, implementation and monitoring. The evaluation should cover aspects such as data management, screening, treatment, referrals, project governance, stakeholder involvement including persons with disabilities, project monitoring, reporting and resource utilisation.

The VIP is multi sectoral and therefore the evaluation will cover all processes implemented in the community, in schools and in target health facilities, as well as with Education Assessment and Resource Centres and the National Council for Persons with Disabilities. In addition, the evaluation will assess how Peek Technology supported the project implementation especially with regards to screening and referral.

iv. Limitations

The evaluation team is likely to face absence of key stakeholders during evaluation period as a challenge.

v. Target audience and learning

The evaluation team will present the findings to the stakeholders including CBM, County Government and partners. The learnings from the evaluation report will be utilised by various stakeholders including County and National Government Ministry of Health, Ministry of Education, Implementing partners Tenwek and Sabatia, Peek Vision, CBM and BMZ.

vi. Evaluation Questions

The questions shall be based on the OECD DAC evaluation criteria of relevance, coherence, effectiveness, efficiency, impact and sustainability.

In addition, all evaluations assess aspects around the CBM Programme Quality Principles (PQF document to be provided to consultants after contract signature), namely:

• Equality and Inclusion
• Gender Equality
• Safeguarding (children and vulnerable adults)
• Advocacy
• Accessibility and Universal Design.

a. COHERENCE – The extent the project aligned with the needs on the ground

• Was the project designed in a way that responded to the needs and priorities of all the stakeholders?
• How well did the project align with current National Eye Health Strategic Plan, County Plans and International priorities?
• To what extent did the project address county-specific eye health priorities?
• How effectively did the project respond to the needs of populations in schools and communities?
• What was the level of community engagement in project design and implementation?
• How well did the multi-sectoral approach aid in the success of the project?
• How well did the project complement existing health programs and systems at national and county levels?
• What synergies were created with other eye health initiatives?
• How well did the multi-sectoral approach aid in the success of the project?

b. EFFECTIVENESS – The extent the project has achieved its objectives.

• To what extent did the project achieve its objectives and expected results as per the design? A critical judgement of the extent of achievements (under- or over- achievement) and reason for the same is expected.
• Detailed assessment of achievements vs. objectives and set targets
• Analysis of success factors and challenges
• Separate performance analysis for Bomet and Vihiga Counties and good practices and learnings
• Assessment of implementation strategies

Training and Capacity Development:

• Evaluation of training of health workers at Kenya Medical training on ophthalmology related courses and relevance to the project goals
• Assessment of knowledge retention and application in achieving project goals
• Analysis of effectiveness of Continuous medical education on quality eye care service delivery

Outreach and treatment activities

• Effectiveness of community and school outreach strategies
• Coverage and reach of treatment outreaches
• Success of community and school screening programs
• Community engagement and awareness creation
• Were there any adaptive approaches in this project? To what extend did they address emerging issues to improve project effectiveness?
• How effectively did the implementing partners coordinate with county governments?
• What mechanisms were established for stakeholder coordination?
• How effective was the public private partnership across all the 7 counties.

c. EFFICIENCY – How well have resources been planned and used?

• Did peek technology improve efficiency in screening and referral pathway compared to the traditional eye screenings methods.
• To what extent did the project deliver results in a cost effective and timely manner?
• Resource Utilization:
• Cost analysis of major interventions
• Compliance with budget and budget lines
• Resource allocation across counties
• Value for money assessment and including use of Peek Technology
• Operational efficiency analysis

d. Time Efficiency:

• Compliance with implementation timelines
• Compliance with procurement processes
• Compliance with administrative procedures
• Response to implementation challenges

e. IMPACT – What difference did the project make?

• To what extent has the project generated significant impact (positive or negative, intended or unintended) higher-level effects?

Human Resources: To what extent did the project contribute to strengthening sustainable human resources for eye health capacity?

• Quantitative and qualitative assessment of capacity development
• Analysis of staff retention and motivation
• Evaluation of skill application and service quality
• Assessment of training cascade effects

Health Information Systems: How has use of technology improved delivery of services while considering below:

• Effectiveness of electronic medical records
• Usage of Peek technology and its impact on the success of the project
• Referral system efficiency
• Default tracking success rates
• Data quality and utilization
• Integration with existing health information systems and the national system

Service Delivery

To what extent has the project improved quality in:

• Outcome of interventions
• Patient satisfaction
• Turnaround time and service efficiency
• Coverage and access improvements
• Integration with primary health care

Access to Essential Medicines and Supplies

To what extent has the project improved the supply chain?

• Availability of essential medicines
• Quality of optical devices
• Stock management efficiency
• Cost barriers and solutions

Health System Financing

Sustainability Analysis: To what extent has the project impacted on:

• Integration into county health budgets
• Cost recovery mechanisms e.g. Facility Improvement Funding model, Insurance
• Insurance coverage for eye care
• Public-private partnerships
• Resource mobilization strategies

Leadership and Governance

Policy Impact: to what extent has the project resulted in;

• Changes in eye health policies
• Integration into health sector plans
• Technical Working Group effectiveness
• Stakeholder coordination mechanisms
• Decision-making processes

f. SUSTAINABILITY – Will the benefits last?

• To what extent are the achievements of the project likely to continue beyond project period? These include but not limited to the financial, technical and institutional capacities of the systems which are needed to sustain quality service delivery.
• Are there any mechanisms in place to support the project achievements in the longer term?
• Did the project influence positively health seeking behaviours on eye care?
• What measures are in place to ensure sustainability of the established institutions with regards to:
• Integration into existing health systems
• Capacity development outcomes
• Policy and regulatory framework
• Stakeholder ownership
• Peek technology
• Buy-in on use of Peek technology
• What measures are in place at the County Government to ensure continuous funding for eye health. Consider measures around budget allocation for eye health and private sector engagement.

g. Safeguarding of children and vulnerable adults

• To what extent were safeguarding aspects considered and how were the respective mechanisms established, used and adhered to during project implementation?
• Were there any challenges encountered of safeguarding nature and what mitigation mechanism were put in place?

h. Equality and inclusion/gender equality

• Have the partners collected disaggregated data according to gender, age (adults/children) and disability status? What was foreseen in the Peek technology towards that and how was Peek technology helpful in collecting and generating such data? How have the partners made use of the results, e.g. in ensuring that all types of stakeholders were equally and appropriately addressed.
• How was disability data collected? Did the partner use Washington Group Short Set of Questions? if not, what were the challenges? If so, how was the data used? If not, what were the other alternatives to data collection to ensure data disaggregation by disabilities.

i. Methodology

The consultant is expected to design the conduct of this evaluation within the given circumstances (geographical, political, time frame) and context of Bomet and Vihiga Counties. During the entire process the below mandatory mechanisms must be adhered to:

• Participation and inclusiveness
• Safeguarding of children and adults at risk
• Data security and privacy (anonymity and informed consent)

A detailed methodology shall be provided as part of the inception report and discussed before implementation. In consultation with the implementing partner, the persons and groups to be interviewed or consulted during the evaluation shall be provided.

In addition, the consultant is expected to specify how stakeholders shall be included in the data collection and analysis, in the formulation of recommendations. It is expected that the findings and recommendations are fed back to them thus the consultant will have to detail specific meetings, discussions, workshops, presentations.

It is expected that the consultant will use a Mixed Methods Approach, incl. qualitative and quantitative assessments.

j. Management Responsibilities and Evaluation Team

Commissioning responsibility

The Country Director will be responsible for contracting. The responsible staff from CBM, including country office staff and the Inclusive Health Initiative Technical Advisor will review and approve the methodology and all final deliverables.

Management Responsibility and Logistics

CBM will take lead in ensuring the consultancy is fully supported. The implementing partners Sabatia Eye Hospital and AGC Tenwek Hospital will assist the consultant with provision of needed project documentation, booking interviews and organizing meeting venues. The partners will coordinate with CBM to identify sign language interpreters as needed. The consultant will be responsible for all logistics arrangements necessary for project site visits and related activities. All costs related to the consultancy will be included by the consultants in the budget including estimated logistics cost.

Evaluation Team

Required core team composition:

• A team leader with experience in health systems evaluation of at least 5 years and considerable experience in development work, public health or community eye health using mixed research methods.
• An experienced eye health professional with program development and implementation experience with special bias to health systems strengthening and management.
• A health economist or finance expert in health financing.
• Demonstrated experience in using various tools and applications in data analysis and presentation.
• Fluent in English and Swahili; speakers of the local language will be a plus.
• Proven track record of successful undertaking of similar assignments.

1. Deliverables

The consultant is expected to deliver the following:

Week 1:

• Inception report including detailed methodology.
• Work plan and data collection tools.

The first payment installment shall be made upon approval of the inception report.

Week 2-6:

• Draft professionally formatted/designed report, incl. thorough analysis of data collected and conclusions.

The second payment installment shall be made upon delivery of the draft report.

Week 6 – 8:

• Stakeholder validation workshop.
• Final report based on feedback from stakeholders.
• PowerPoint presentation of the findings and recommendations/lessons learnt.

The report should be presented in the standard evaluation report format of CBM (max. 25 pages) plus annexes and with at a concise summary (max 3 pages) in English. Please note that all background material shall be delivered, including questionnaires, surveys (in anonymised form) etc. in the form of an Annex.

The consultant must confirm that the reported work is their own personal intellectual creation and that they have not created substantial parts of the deliverables by using AI systems.

The final payment installation shall be made upon approval of all deliverables.

1. Evaluation Schedule

The evaluation will take 46 days details to be discussed later

Tagged as: CBM Kenya, Kenya