South Sudan: Evaluation of UNHCR South Sudan Livelihoods Strategy 2016-2018

Nov 19, 2018 | Relief International
View Original

Introduction

UNHCR South Sudan representation in 2016 launched a three year medium term livelihoods strategy to provide overall strategic direction to partners supporting livelihoods programming in key refugee locations in South Sudan. The strategy was aligned to and designed to complement interventions aimed at achieving positive protection outcomes for refugees and other persons of concern. The strategy sought to contribute to socioeconomic self-reliance through targeting 70% of refugee households in protracted refugee locations (Jonglei, Central and Western Equatoria) and 30% refugee households in emergency situations (unity and upper Nile). Over the course of 2016 to 2018 strategy implementation, UNHCR has supported partners (Relief International and ACTED) in the implementation of key interventions in the refugee camps and the hosting community In Maban County, Upper Nile State. The interventions undertaken largely comprised of: agriculture livelihoods; economic recovery – self-employment/ income generation activities and; skills development –Skill based and vocational trainings. The strategy was launched at a time when market driven livelihood programming was being promoted as means to promote greater economic inclusion and local economic development. With the medium term strategy implementation period coming to a close in 2018, there exists increasing interest in evidence and results realised post implementation aligned with specific objective to promote socioeconomic self-reliance as well as overall protection and solutions for refugees.

As from 2015, UNHCR globally has been promoting a market systems approach to livelihoods programming requiring design of livelihoods programming to be market driven and market aware. The current decentralised impact evaluation is timely and relevant as it occurs within an ecosystem of changes in strategic direction aimed at promoting a new way of working that supports refugees self-reliance based on improved livelihoods, expansion of economic opportunities, greater economic inclusion and support to local economic development for hosting countries through responsibility sharing. Events within the ecosystem have included development of Comprehensive Refugee Response Framework aligned to Global Compact for Refugees; Multi-Year Multi-Partner Protection and Solutions Approach, UNHCRs involvement in the revision of the Minimum Economic Recovery Standards through SEEP Network Partnership and launch of Refugee Livelihoods and Economic Inclusion strategy concept note among other strategic collaborations.

The 2016 to 2018 livelihoods strategy was implemented in key locations hosting refugees clustered in ten camps, settlements and locations in Upper Nile, Unity, Central Equatoria, Western Equatoria and Jongolei states. At the time of drafting the strategy, as of 31 December 2015; the population of refugees was 263,000 with a profile of 36.37% adults; 39.65% children; 21.46% U5s; 2.52% elderly of 60+ years and an estimated average household size 5. The refugee population keeps increasing due to new arrivals mostly displaced as a result of ongoing conflict in South Kordofon and Blue Nile state in Sudan. Refugees population as of October 2018, is approximated at over 300,000 individuals (69, 686 total households) and an estimated household size of 6. Sudanese refugees at 93% comprise the largest of all nationalities hosted in South Sudan, followed by Democratic Republic of Congo (5%), Ethiopia (1%), and Central Africa Republic (1%). Sudanese refugees are currently hosted in Unity and Upper Nile. Current impact evaluation will be undertaken in Upper Nile, Greater Maban which hosts approximately 144,000 individuals in four refugee camps namely Doro, Yusuf Batil, Kaya and Gendrassa camp where livelihood interventions have been supported as from 2016 to date.

The impact evaluation is expected to generate impartial and credible evidence to inform strategic direction as from 2019 and beyond. The evidence generated will be used as a baseline for updating 2019 to 2022 refugee livelihoods and economic inclusion strategy for South Sudan.

Specific Context Overview

Armed conflict, political instability, localized host community-refugee tensions coupled with adverse effects of climate change  in neighbouring Sudan and within South Sudan; has contributed to massive displacement of approximately 144,000 refugees currently  settled in four refugee camps namely – Doro, Yusuf Batil , Kaya and Gendrassa in greater Maban, upper Nile of South Sudan. Refugees, host community and other displaced persons find themselves in precarious socio-economic situation with many reporting food shortage as a major concern. The major source of food for refugee households is food aid distributed through World Food Programme at 30% reduced ration as from August 2015.

Markets continue to function poorly in Maban and procurement of goods as well as services locally is significantly limited. Market growth has been largely compromised by poor market structures to facilitate trade, limited skills set, weak entrepreneurship base, limited financial capital and economic shocks owing to hyperinflation and devaluation of the South Sudanese pounds.  Refugees have limited access to external markets in Maban; water scarcity, erratic rainfall, increased incidence of pests and disease continue to negatively impact agriculture yield. Limited start-up capital and access to finance remains a major constraint for refugees and vulnerable host community to be able to meaningfully participate in the local economy.

Through UNHCR South Sudan National Livelihoods strategy (2016-2018) and aligned to UNHCR’s global strategy for livelihoods (2014-2018); UNHCR with key livelihoods implementing partnership have supported Agriculture livelihoods, economic recovery and skills development interventions in the refugee camps of Maban. It is against this background that UNHCR jointly with implementing partnerships has commissioned the impact evaluation for accountability and learning as strategy implementation period lapses.

Purpose and Objectives

The impact evaluation of UNHCR South Sudan 2016 -2018 Livelihoods strategy will serve a dual and mutually reinforcing learning and accountability purpose.The purpose of the decentralized evaluation as commissioned jointly by UNHCR and Implementing partnership is to contribute strategic and timely evidence on impact to not only inform update of 2019 to 2022 refugee livelihoods and economic inclusion strategy of South Sudan; but also contribute key lessons learned to inform best practices on refugees economic inclusion.

The primary audience is UNHCR and Implementing Partnership senior management, Livelihoods units, Programme and Protection Unit as well as other relevant sector leads. Secondary audience include other humanitarian, development and relevant private sector actors engaged in promoting refugee livelihoods and economic inclusion.

Impact Evaluation Approach

Scope

• Population: The impact evaluation will include refugees and other persons of concern targeted by livelihood interventions implemented as from 2016 to 2018
• Time frame: The impact evaluation will assess key livelihood interventions implemented as from 2016 to 2018
• Location: The Refugee camps of Maban – Doro, Yusuf Batil, Gendrassa and Kaya camps , Upper Nile

Key Impact Evaluation Questions (KIEQs)

The impact evaluation will seek to answer key questions (KIEQs) highlighted. The analysis need to answer the KIEQs will be based on feedback from sub-questions and will be further refined during the impact evaluation inception phase.

KIEQ1: How relevant/appropriate was the context specific strategy [through key livelihood interventions implemented] in promoting socio-economic self-reliance and improving overall protection for targeted population?

KIEQ1 will seek to answer the sub-questions highlighted below:

• Were the key livelihood interventions [Agriculture livelihoods, Economic recovery and skills development] promoted consistent with priority needs of the targeted population?
• Were the activities and outputs consistent with objectives and overall intended impact?
• What were some of the relevant protection risks and vulnerability the livelihood strategy responded to? How did the strategy contribute to improving protection of targeted population?
• Was their sufficient institutional capacity to implement relevant and appropriate livelihood programming aligned to socio-economic self-reliance objectives? What are some of the key lessons learnt and recommendations to improve future programming?

KIEQ2: To what extent did the livelihood interventions achieve strategy objectives?

KIEQ2 will seek to answer the sub-questions highlighted below:

• What key factors contributed to achievement of strategy objectives? What key factors contributed to failure to achieve the objectives?
• What coordination arrangements were in place for achievement of the objectives? How did the coordination arrangement contribute to achievement of the strategy objectives?
• Were available resources channelled in a coordinated manner towards achievement of the strategy objectives?

KIEQ3: How efficiently were available livelihood resources utilized in achieving results?

KIEQ3 will seek to answer the sub-questions highlighted below:

• How were financial, human and material resources available for the strategy implementation converted into impact?
• How cost-effective were the project activities when compared to results realized?
• How efficient was the context-specific implementation modality chosen compared to other relevant modalities?
• What is the correlation between a household receiving general food ration distribution and participation in livelihood interventions promoted by 2016 to 2018 strategy?

KIEQ4: What is the impact of UNHCR South Sudan 2016-2018 Livelihoods Strategy?

KIEQ4 will seek to answer the sub-questions highlighted below:

• To what extent did Agriculture livelihoods, economic recovery and skills development interventions supported in 2016 -2018 contribute to socioeconomic self-reliance? How did the specific interventions contribute to self-reliance of targeted populations?
• What key factors contributed to impact reported? What key challenges/constraints curtailed progress towards achievement of impact?
• For whom has the greatest positive impact been felt and why?
• Did the implementation of the strategy result in unintended positive and negative impacts? What are some of the positive and negative unintended impacts?
• What key lessons and best practices can be derived from impact realized? What are some of the examples of pilot interventions that can be replicate elsewhere in South Sudan? [ Document and annex at least 3 success stories/ case studies following strategy implementation]

KIEQ5: What is the sustainability of UNHCR South Sudan 2016-2018 Livelihoods Strategy implementation?

KIEQ5 will seek to answer the sub-questions highlighted below:

• What are some of the benefits realised during 2016-2018 livelihood strategy implementation that are likely to continue post 2018? And also over the long term should donor funding cease?
• Were longer term considerations taken into account during the design of the UNHCR South Sudan Medium Term Strategy design and also in implementation of key livelihood interventions? What are some of the key recommendations for future livelihoods programming on longer term considerations to be explored?
• Were partnerships leveraged to support continuity of results over the long term? What are some of the strategic partnerships that are key – currently and in future?
• To what extent were local capacities supported and developed? Elaborate with key recommendations?
• What key factors are responsible for achievements and failures of the livelihoods strategy sustainability? How can the factors be strengthened or mitigated?
• Based on humanitarian assistance provided particularly general food distribution, what are some of the strategies that could potentially be employed to ensure targeted population do not remain in perpetual dependence of humanitarian assistance? What are some of the potential exit strategies taking into account the 2016 to 2018 UNHCR South Sudan livelihoods strategy?

4.3   Approach and Methodology

• The impact evaluation methodology should make use of combination/ mixed methods approach employing quantitative and qualitative methods to answer the 5 Key Impact Evaluation Questions along with the Sub-questions. In summary, the evaluation will comprise:
  • Secondary data review (Desk review and content analysis of relevant contextual information, programmatic data and project documents);
  • Primary data collection - Key informant and in-depth interviews with UNHCR and Implementing Partnership staff (specifically livelihoods, programme and protection unit staff), Operational Partners and where relevant key inter-agency stakeholders [OCHA, UNDP,FAO], local authorities and focal points in government line ministries; Field data collection in the refugee camps – Rapid household surveys, structured interviews and focussed group discussions with refugees/ persons of concern targeted /who benefited from UNHCR supported livelihoods programming during the livelihoods strategy implementation period as from 2016 -2018
• UNHCR welcomes the use of diverse, participatory and innovative evaluation methods and approaches. The impact evaluation is thus expected to employ a robust mixed-method approach incorporating qualitative and quantitative data collection and analysis, including analysis of monitoring data. Qualitative methods should include observation, interviews and focus group discussions (FGD) with a range of key stakeholders including UNHCR and Implementing Partnership Staff at country-level as well as field locations where the refugee camps are located, national authorities and targeted populations with Age, Gender and Diversity (AGD) approach. Data/ information from a wide range of sources will need to be triangulated and cross validated to ensure the credibility of evaluation findings and conclusions.
• The evaluation will draw upon information and analysis collected from a wide range of sources and a representative range of stakeholders. The UNHCR and implementing partnership Evaluation Managers will ensure the Evaluation Team has access to relevant documents and personnel, and will assist in the organization of field missions. The Evaluation Team (ET) is expected to refine the methodology and evaluation questions following the initial desk review and key informant interviews during the inception phase. The final inception report will specify the evaluation methodology, the refined focus and scope of the evaluation, including the Key Impact Evaluation Questions the sampling strategy and the data collection instruments.
• Overall, impact evaluation methodology is expected to:
  • Reflect an Age, Gender and Diversity (AGD) perspective in all data collection activities carried out as part of the impact evaluation – particularly with refugees
  • Employ a mixed-method approach incorporating qualitative and quantitative data collection and analysis tools including the analysis of monitoring data –as available
  • Refer and make use of relevant internationally agreed evaluation criteria such as those proposed by OECD-DAC and adapted by ALNAP for use in humanitarian evaluations
  • Refer to and make use of relevant sectoral standards and livelihoods-specific analytical frameworks
  • Be based on analysis of (i). Strategy and operational guidelines underpinning livelihoods programming being evaluated; and (ii). Main actors and stakeholders
  • Make use of a wide range of data sources for analysis including but not limited to [Livelihoods monitoring data, mission reports, coordination group meeting minutes, livelihoods strategy narrative, indicator narrative reports, mid-year and year-end review reports, relevant assessment reports] in order to demonstrate impartiality of the analysis, minimise bias, ensure credibility of the evaluation findings and conclusions
  • Be explicitly designed to address the Key Impact Evaluation Questions – taking into account evaluability, budget and time constraints

IMPACT EVALUATION TEAM QUALIFICATIONS

The impact evaluation will be undertaken by a dynamic and diverse team of consultants selected through a competitive bidding process. It is anticipated that the team will consist of no less than two people for the duration of the consultancy assignment. Bids for the contract should designate a Team Leader for the project. The consultants should be able and willing to travel to the field locations in Bunj, Upper Nile; South Sudan

Impact Evaluation Team Leader

• A post-graduate degree in economics or social sciences plus a minimum of 10 years of relevant professional experience in food security and livelihoods programming in humanitarian and development context.
• Minimum of 5 years of Impact Evaluation experience in humanitarian and /or development contexts
• Proven experience in successfully leading an evaluation team and managing field data collection in complex , fragile and conflict affected settings
• Technical expertise or wide experience in refugee protection work and livelihoods including a good understanding of relevant analytical frameworks and programming approaches
• Knowledge and understanding of UNHCR protection mandate and modus operandi
• Proven experience and skills with various data collection, analytical methods and techniques used in evaluation and operational research
• Experience in generating useful, credible evidence and action oriented recommendations for accountability and learning post impact evaluation
• Strong written and verbal skills in English
• Excellent technical report writing skills

Evaluation Team Member

• University degree in economics and social sciences (relevant areas include: microfinance, agriculture, food security and livelihoods), plus a minimum of 5 years professional experience relevant to refugee protection, food security and livelihoods.
• Proven experience and skill set in undertaking quantitative and qualitative data collection and analysis for impact evaluation purposes.
• Good understanding of measuring vulnerability and protection risks including use of relevant analytical framework and livelihoods programming approaches
• Well versed in impact evaluation approaches, with a good understanding of humanitarian and development contexts
• Strong facilitation and communication skills – verbal and written English