Preconditions for an Impact Evaluation in Myanmar
Having been ruled by a military regime for more than two decades, Myanmar (formerly Burma) has taken some initial steps along a path of democratisation that began in 2011. In view of these changes, a number of multilateral and bilateral donors, including Germany, have decided to resume the development cooperation with Myanmar that had been suspended many years ago. While first Technical Cooperation (TC) measures have already begun in 2012, additional Technical and Financial Cooperation (FC) interventions started in 2014 and 2015. Both the TC and FC inter alia focus on the priority area of Sustainable Economic Development, specifically on the sectors of vocational technical education and training, private and financial-sector development, and quality infrastructure.
In donor and recipient countries of development cooperation, an increasing interest and call for impact evaluations could be noticed during the last decade. However, the quality and robustness of information provided through an evaluation greatly depends on the data available and subsequently the design and methods that can be used. High quality and rigorous impact evaluation is thus only possible when evaluation and data collection were embedded into the programme cycle from the very beginning, assuring that certain preconditions are in place, including a coherent and well documented programme logic and baseline and monitoring data.
Tasks of DEval
With the resumption of German-Myanmar bilateral cooperation in 2012, the German Ministry of Economic Cooperation and Development (BMZ) saw this as a unique opportunity to pay special attention to impact orientation of the German-Myanmar Programme on Sustainable Economic Development. Consequently, DEval was asked to incorporate evaluative thinking and evaluation expertise into the overall planning process of the programme with the aim of creating the preconditions for an impact evaluation on programme level at the beginning of implementation. First, DEval advised and supported BMZ and the involved implementing organisations in the development of a programme theory, including results models on project level, to create a coherent understanding of expected results and what activities and causal mechanism shall lead to certain anticipated changes. It then was considered through which indicators these results can be measured. Second, a longitudinal design was chosen for the implementation of the impact evaluation, starting with a baseline study to ensure systematic collection of comparable data before, during and after programme implementation. These data will also help providing information when decisions need to be taken.
Since all data collection instruments are to generate panel data according to the longitudinal design, follow-up data collection takes place in 2016 and 2017. After implementation of the German-Myanmar programme on sustainable economic development in 2019, DEval will conduct the impact evaluation, during which the baseline study will serve as basis for comparison.
Furthermore, lessons can be generated from experience and become valuable to the task of improving standards for the results-oriented planning of development cooperation projects and ensuring that the right conditions are put in place for conducting impact evaluations.
DEval’s input involves the following activities:
- Supporting the project teams in the task of developing results models and results-based indicators.
- Performing a comprehensive baseline study (and follow-up surveys).
- Providing partners with training and capacity building in the field of M&E.
Elements of the baseline study and where we are standing right now
- Expert Panel: 59 qualitative expert interviews have been conducted between late 2013 and early 2015 in the fields of private and financial sector development as well as TVET and quality infrastructure.
- Banking Study: In May-2014, a standardized, quantitative survey amongst 57 management representatives from all 19 national banks has been conducted.
- Small and Medium Enterprise (SME) Survey. The first phase of the SME survey (including a block screening to compile a complete list of firms in certain areas, which was necessary due to the incompletion of existing registers, pre-test of questionnaire, etc.) has been completed in November 2014. During the main phase, which took place between December 2014 and March 2015, around 2,500 SME were surveyed in 11 cities. Results from this survey are compiled in DEval’s SME Survey Report, which was published in October 2015.
- Results from Expert Panel, Banking Study and SME Survey are compiled in DEval’s Baseline Report for the German Myanmar Programme on Sustainable Economic Development, which was published at the beginning of 2016.
- Follow-up data collection is planned from late 2016/2017 and the final impact evaluation is scheduled for 2019 after implementation of the country programme.