Final Evaluation Study - Empowering Civil Society Programme

1. Context

• The Project


Search is implementing a 18-months project funded by the European Union (EU) with the objective to “Advance the role of civil society organizations (CSOs) to lead sustainable, conflict- and gender-sensitive, and equitable stabilization and social cohesion of target communities”. The project  strengthened registered and unregistered CSOs’ capacity to address community needs related to stabilization and social cohesion in an inclusive, participative, and transparent manner.


The project focused on building the capacity of civil society actors as reliable partners in local development processes, proposing innovative ideas and approaches which address community challenges and promote sustainable and positive change in a participative manner in three target areas.


The CSOs also engaged with diverse community stakeholders such as community leaders and other professionals, with a specific effort to include women, youth, PWD, and other vulnerable groups to represent the diverse constituencies, utilize tools to break down barriers of prejudice and stereotypes between and among groups, implement mechanisms to ensure participation of community members with different needs, and promote a culture of social cohesion within communities. Through participation in the project activities, community stakeholders engaged in an inclusive and transparent process to enhance stabilization and social cohesion and build trust through collaborative action.


Search conducted an initial mapping to identify CSOs through specific selection criteria. The results of this intervention realized a successful implementation of capacity building workshops, participatory community stakeholder mapping, community dialogues, collaborative action plans, community committee meetings, small grants scheme, coaching sessions and shared learning workshop in each target area, which provided the CSOs with a key opportunity to apply skills gained and contribute to achieving the project’s objectives.


Target groups  


The primary target groups of the project are 12 CSOs and six community committees including 62 community stakeholders. The secondary target groups are the wider members of target communities; specifically vulnerable women groups such as internally displaced persons (IDPs), minorities, and poor households.


The group of 12 CSOs, community committees and stakeholders are distributed across the three areas.

2. Goal and Objectives of Study

• Search is seeking a consultancy team/firm to conduct a summative Endline Evaluation study to understand the extent to which the project has contributed in advancing the role of  CSOs to lead sustainable, conflict- and gender- sensitive, and equitable stabilization and social cohesion in target communities. This study will also aim to assess the project’s progress towards its specific objectives and considering the remote implementation:


• SO.1: Strengthen the capacity of selected CSOs to advance sustainable and positive change in stabilization and social cohesion in target communities; and
• SO.2: Promote cooperation between CSOs and key stakeholders across divides to lead stabilization and social cohesion programs that are responsive to target communities’ needs.


The Final Evaluation Study should enable Search and donors to inform future programme design and will assess the overall success and effectiveness of the implemented project. In specific terms, The Final Evaluation has the following objectives:


• Assess the overall impact of the project based on evidence-based outcomes whether intended (according to the project’s logical framework) or unintended outcomes (positive or negative) and determine Search’s contribution to identified changes;
• Update the project’s log frame and indicators with endline values;
• Assess whether the project had produced the effects as envisioned in the Theory of Change in addition to assessing the project’s Relevance, Efficiency, Effectiveness and Impact, and Sustainability as per the Organization for Economic Co-operation and Development - Development Assistance Committee (OECD-DAC) evaluation criteria in addition to the overall project’s adaptability; and
• Capture good practices, compile lessons learned throughout the lifespan of the project and draw actionable and evidence-based programming and policy recommendations to inform future programme design in a similar context.

3. Key Questions of the Study

• End-line Questions
The analysis will be focused on the following evaluation questions organised per OECD-DAC criteria:


Relevance:
• To what extent did the intervention’s objectives and design respond to the needs of CSOs in strengthening their roles to lead sustainable, conflict- and gender- sensitive, and equitable stabilisation and social cohesion of target communities?
• To what extent has ‘do no harm’, conflict sensitivity and barriers been taken into consideration in the design and implementation of community-led initiatives and what are the effects of these interventions and how are they contributing to peacebuilding?
• To what extent did the intervention’s objectives and design align with and contribute to  the plans of target CSOs,  Search’s Levant Strategy and EU plans in the target country?


Effectiveness and Impact:
• To what extent has the intervention been effective in meeting the essence of the Theory of Change and achieving its results and planned outcomes by addressing common issues and needs of targeted communities- considering the remote implementation, by assessing:
• The level of change in knowledge and capacities of CSOs;
• The level of change in personal agency among CSOs towards the ability of making positive difference in the community around conflict and gender issues;
• The change in CSOs and community stakeholders' perception of having tangible opportunities to cooperate with one another to promote stabilization and social cohesion in target areas; and
• The level of engagement and coordination between CSOs and key stakeholders across divides to lead stabilization and social cohesion programs that are responsive to communities’ needs.
• Has the project been able to engage the CSOs with community stakeholders, including PWDs in their efforts to advance stabilization and social cohesion?  
• What unintended outcomes (positive and negative) emerged from project implementation?
• What factors (positive and negative) have had the greatest influence on the achievement of results?
• To what extent did the intervention contribute to longer-term effects (impact)?
• Was the level of partnership(s) with CSOs and the implementing partner appropriate to support the effective achievement of the intervention’s objectives?


Efficiency and Value for Money:
• To what extent have resources been allocated and utilized in an efficient manner to achieve value-for-money?
• How efficient was the delivery of the intervention in terms of expenditure and implementation of activities?
• How efficient was the remote implementation approach and through partners  towards the achievement of the objectives? How could this remote approach be improved?
• What was the return on investment attained?


Sustainability:
• To what extent are the benefits of the programme likely to be sustained in the long term in terms of 1) CSOs capacity building, and 2) CSOs’ collaboration and engagement with communities and key stakeholders to promote stabilization? What are the main factors behind this?
• Are the systems, policies, processes and practices built/improved under the project likely to continue in use or benefitting the CSOs?
• To what extent key project activities already have multiplier effects that extend beyond the targeted CSOs?
• What are efforts required to improve and strengthen sustainability?
• To what extent did the CSOs show efficiency and ownership towards the project?


Lessons Learned:
• What lessons can be learnt or good practices can be identified from the implementation of the project? This to be supported by evidence.
• What notable programming and policy recommendations can be drawn from this project?

4. Geographic Locations

• The Endline Evaluation study will provide quantitative and qualitative information, gender, age, ethnicity, religious background, and residency status (host community member, IDP and refugee) disaggregated data and analysis in a strategic county in the Levant region. 

5. Methodology and Data Collection Tools

• The Endline Evaluation Study will utilise a mixed method approach consisting of both qualitative and quantitative primary and secondary data collections. The study should triangulate data from multiple sources and stakeholders in order to infer reliable findings.


Desk review: the consultant(s) will review key project documents including the proposal, the baseline and mapping report, MEL framework, logframe and other related monitoring and programmatic documents  and set meetings with key personnel to better understand the project, its goal and activities. The consultancy firm is expected to review the above background documentation as part of the desk review phase of the study, in addition to the literature review about the role of civil society organisations and the conflict dynamic in the targeted areas. Also, the consultant will utilise primary available M&E data that was formerly collated by project staff including activity surveys and assessments, baseline data, project quarterly reports, etc.


Data collection tools and approach: The consultant(s) will develop quantitative and qualitative data collection tools and collect data in the form of surveys, interviews, focus group discussions (FGDs), and possibly case studies using a solid sampling approach. To ensure a better understanding of the key issues that will be addressed in the project, the data will be disaggregated by age, sex, gender, age, ethnicity, religious background, and residency status  (host community member, IDP and refugee), persons with disabilities as well as by geographical location.


Sampling strategy and approach: The consultant(s) shall determine the appropriate sample size and approach in consultation with Search MEAL Manager and referring to the M&E plan. The sampling strategy will take into consideration the activities carried out, target population and the project areas (95% Confidence Level and 5% Margin of error). The consultant(s) will meet with the project participants, partners, Search staff and relevant stakeholders.


Evaluation matrix: The consultant(s) is also required to develop a detailed evaluation matrix with judgment criteria that captures the study’s objectives and evaluation questions of which will guide the evaluation process into reliable judgments and outcomes.


Search’s MEAL Manager will validate and approve the final methodology and tools before data collection begins, as well as provide feedback on the report.


Data collection, analysis and the results should represent Inclusiveness—the methodology should include a wide range of viewpoints, specifically gender and age-sensitivity when applicable.

6.Deliverables

• Search expects the following deliverables from the external consultant(s) as they correspond to the timeline and budget:


• An inception report detailing the methodology, data collection tools, analytical framework, approach and timelines;
• All original datasets should be submitted to Search (Excel format for the quantitative data and all transcripts for the qualitative data); In addition to any necessary training of data collectors or set-up of systems for data collection;
• A draft endline report in English for review by Search staff; and
• A final report in English (40 pages max in length, excluding appendices) consistent with Search branding and standards for evaluation. The report will:
• Uses the Search reporting template;
• Provides a clear connection between the conflict or context assessment and the intended results, articulate the project’s Theory of Change (ToC), and include other relevant project specifics;
• Fully explains the objectives and research questions of the study, limitations and methods chosen for analysis; 
• Findings respect Search’s evaluation standards, are structured around the main objectives of the study, and are presented in relation to the intended target groups.
• The findings should speak to the link between the project, its ToC and its contribution to our strategy. It should explain adaptations that occurred during the project and their impact on results;
• Recommendations should have a clear audience and be specific, accessible, and actionable;
• Indicator table showing all indicators; and
• Appendices should include detailed research instruments, list of interviewees, terms of references and evaluator(s) brief biography.
• A two-pager summary highlighting key lessons learnt and programming and policy recommendations  from this project that will inform future Search and donor programming; and
• A PowerPoint presentation that will be used by the consultancy team to present the findings to Search and the donor after the final report is approved.

7. Logistical Support

• Regardless of the modality chosen, the firm/consultant(s) should be fully responsible for the data collection, including working with data collectors to be recruited and trained by them in case needed. In case of in-person data collection, the firm/consultant(s) will be responsible for organizing their own logistics for data collection (vehicles, fuel, and  drivers). All of this must be budgeted into the financial proposal.


In addition, Search and partners will share the following elements with the external consultant(s): background materials including the project proposal and log frame, etc.

8. Ethical Considerations

• The consultancy should take into consideration  the rights of participants in line with the ethical codes and guidelines of research. The participants should be asked to provide their consent and made aware that their identities would not be revealed. Cultural rights will be respected. The consultant shall remain alert and responsive to any child safeguarding risks, acquire relevant knowledge and skills which will enable promoting strong safeguarding practices, understanding the child safeguarding policy and procedures.


The firm/consultants are required to respect the following ethical principles:
• Comprehensive and systematic inquiry: Consultant should make the most of the existing information and full range of stakeholders available at the time of the review. Consultant should conduct systematic, data-based inquiries. He or she should communicate his or her methods and approaches accurately and in sufficient detail to allow others to understand, interpret and critique his or her work. He or she should make clear the limitations of the review and its results.
• Competence: Consultant should possess the abilities and skills and experience appropriate to undertake the tasks proposed and should practice within the limits of his or her professional training and competence.
• Honesty and integrity: Consultant should be transparent with the contractor/constituent about: any conflict of interest, any change made in the negotiated project plan and the reasons why those changes were made, any risk that certain procedures or activities produce misleading review information.
• Respect for people: Consultant should respect the security, dignity and self-worth of the respondents, program participants. Consultant has the responsibility to be sensitive to and respect differences amongst participants in culture, religion, gender, disability, age and ethnicity.


Applicants are kindly required to integrate a detailed multiple scenario strategy in their proposed methodologies and work plans that would ensure efficacy in the delivery of the assignment and allow for flexibility to switch between different modalities (i.e., to virtual data collection and vice-versa) whenever there is a need to do so.

9. Data Quality Assurance and Management

• All deliverables will be reviewed and approved by the country office and global Institutional Learning Team (based in Brussels) prior to acceptance of the final product. The consultant(s) are also required to develop a clear and rigorous quality control plan during the inception phase of which will be implemented and deployed throughout all phases of the evaluation process to ensure reliability and validity of findings.


To ensure effective data management, all the qualitative collected data (through FGDs, KIIs) will be transcribed with the respect to the data collection ethics. All the quantitative surveys will be cleaned and stored in a secured database. Access to the data will be allowed to the staff members who work on the final evaluation and will be limited to those who have the password.

10. Timeline

• The evaluation timeline is expected to be kicked off in mid-April 2023 and concluded by the end of June 2023 by submitting the final deliverables requested above.

11. Budget

• The total budget available for this study is 16,000 USD. A detailed budget should be provided, including daily rates for personnel, and costs related to data collection (per total number of people sampled, sites for collection, etc.), analysis, and production of deliverables. The selection committee will give preference to the applicant(s) with a relatively lower budget and higher quality technical proposal.

12. Requirements of Consultant

• Education:
• Lead consultants must have at  least a Master’s degree (PhD preferable) in research methods and/or evaluations, development, conflict or peacebuilding, M&E or any related social sciences.


Skills and experience:
The following skills and experience are expected by Search for our evaluator for this project:
• Excellent proficiency in English and Arabic - written (including professional English reporting) and spoken;
• More than five years of experience in programme/project evaluation, including collecting data in interviews, surveys and focus groups and analysis;
• Experience  in carrying out studies, evaluations, and baseline on Civil Society Support Programmes in the Greater Levant region countries such as Iraq, Syria and Lebanon.
• Experience applying remote data collection  modalities in conflict areas in the Greater Levant region;
• Experience in conflict analysis and working with justice, civil society sectors and gender related topics;
• Experience in doing evaluations for peacebuilding and democracy, human rights and governance programmes in MENA region;
• Experience working with international organizations;
• Experience conducting quantitative surveys and analysis;
• Monitoring and Evaluation methods and data collection skills; and
• Familiarity and experience with contextual challenges in the geographic location(s) where the study will take place.

13. Selection Criteria

• Consultant proposals will be selected for:


• Relevance of proposed methodology to the goal, objectives, and research questions for the project.
• Quality of proposed methods, conflict sensitivity approaches,and quality control measures.
• Qualifications of the candidate(s).
• Proposed budget in relation to proposed methodology, deliverables and team.
• Timeline for proposed activities.

14. Applications

• To apply, interested candidates (individuals or teams) are requested to submit the following documents:


• A Technical Proposal clearly outlining the proposed methodology, understanding of requirements, experience doing similar work, and timeline for the endline together of not more than ten pages (excluding annexes); this to be combined with:
• Short cover letter (not more than one page) with at least one writing sample (preferably an endline evaluation); and
• Curriculum vitae of lead consultant(s) - consolidates in one pdf.
• A Financial Proposal (with detailed line items per deliverable) for the completion of the aforementioned deliverables (not more than two pages).