WILL THE EXPECTATIONS FROM THE ROADMAP BE REACHED?
07.03.2018 / How should the monitoring and evaluation methodology of the Strategic Road Maps of Azerbaijan look?
The drop in crude oil prices in the global market since 2014 has worsened the macroeconomic and fiscal parameters in Azerbaijani economy. Much like other oil-dependent economies, the global decline in oil prices has negatively affected Azerbaijani economy. Revenue from oil constitutes more than 60% of the state budget, while the share of oil and gas exports constitutes 88% of total exports. Given the severe dependence on oil and gas exports, Azerbaijani economic growth fell rapidly in a short period (Chart 1).
This economic reality, based on a number of parameters, could have been evaluated as a "recession." There are two different approaches in economics to "recession" phenomena:
a) Technical recession: the decline of GDP over two consecutive quarters (negative economic growth). In 2017 economic growth in Azerbaijan was negative in all quarters and months (except for January and February).
b) Recession as an element of the business cycle: in these circumstances, all the economic indicators (industrial production, unemployment, sales, real income, etc.) deteriorate.
The majority of the indicators in Azerbaijan listed above in 2015-2017 has deteriorated. Thus, the Azerbaijani economy declined by 3.1% in 2016, while economic growth in 2017 was only 0.1%. Inflation has reached a two-digit number: 12.9%; industrial production has dropped 3.4%. The national the currency is worth half as much as it did before, compared to the USD, and the ratio of external public debt to GDP has reached 19%.
This crisis period for the Azerbaijani economy has compelled the government to take some steps. As a result, the Azerbaijani government adopted 12 strategic roadmaps encompassing 11 economic sectors and the national economy in general based on the decree the president issued on December 6, 2016. "Strategic Road Map on National Economy Perspective" with a focus on economic development covers the short, medium and long-term periods. The strategic roadmap comprises economic development strategy and action plan until 2020, long-term vision for the period up to 2025, and targeted vision for the period beyond 2025.
The Center for the Analysis and Communication of Economic Reforms (CACER) that was founded on April 20, 2016, within the frameworks of "Post-oil period" reforms have already released half a year monitoring and evaluation reports of the Strategic Road Map for 2017. The monitoring and evaluation report reflects the measures that have been taken on the implementation of the strategic roadmap, as well as some of the issues and recommendations.
First of all, the periodic release of monitoring and evaluation reports on the implementation of Strategic Road Map is a positive step regarding transparency and public accountability. However, it is important to enhance the quality of this reporting process and adjust it to the international best practices.
Legal-institutional direction
In accordance with the "Rules of the state bodies submitting requests to CACER and financing" that was adopted on November 30, 2016, by presidential decree, the center conducts risk assessment (provision 2.0.1), and monitoring and evaluation (provision 2.0.2) of state programs on economic development based on the requests of various state bodies. However, it is possible to add to the 2.2 provision ("Duties") of the statute of CACER: "The center prepares monitoring and evaluation methodology and reports on long-term development strategy of state and national programs (sectoral and regional). The statue of Ministry of Economy despite having the provision (2.0.17 provision) on "coordination and monitoring of the concepts, and preparation of national programs on social-economic development, including priority areas," lacks a note on designing monitoring and evaluation methodology. There is also a "Monitoring and Oversight of national programs" office functioning within the department on regional development and national programs in the ministry. Long-term economic development strategies and monitoring and evaluation methodology of national programs can be prepared in collaboration of CACER and the sector mentioned above of Ministry of Economy by the authority granted to them by their respective statutes and charters.
Monitoring and Evaluation methodology
The analysis of OECD documents and the experience of post-Soviet countries demonstrate that the monitoring and evaluation of economic development strategies and national programs are more effective when it is based on well-designed methodology. "The rules of Preparation, Implementation, and Monitoring and Evaluation of State Investment Program" that was adopted on March 17, 2010, lacks monitoring and evaluation methodology, while this document cannot sufficiently regulate the monitoring and evaluation of the implementation of Strategic Road Map.
The monitoring and evaluation methodology of the strategic roadmap on the national economy and eleven areas of the economy must be based on the verifiable system of indicators. It is essential to come up with the outcome and impact-oriented indicators which would monitor and evaluate the effectiveness and impact of the strategic roadmaps. The indicators need to be able to ring an early alarm on either progress or stagnation within the frameworks of "SMART" goals and objectives of the roadmap. These indicators would ensure feasibility of the activities that will be taken based on the action plan of the strategic roadmap. The monitoring and evaluation mechanism that is described in Table 1 following international best practices can be utilized for effective monitoring and evaluation purposes. The table lists goals, general outcome, real observable outputs and more specific activities concerning every objective or action plan. The most critical element in this methodology are the indicators, definition of indicators, current and targeted value, and as well as the mentioning of sources of data. Risks, hypothesis, and means of verifying the information can also be added to this list.
Analysis and recommendations on CACER's monitoring and evaluation reports
The 2017 half a year monitoring and evaluation reports of the Strategic Road Map released by CACER has a few important points that could be improved in following ways:
1) CACER, while evaluating the performance of the main implementing actors, "has given 1 point to the activities that have been implemented, 0.5 points to the activities implementation of which has started, and 0 points to the activities implementation of which has not started at all and then divided the total sum of the points to the total number of activities that the implementing actors had to implement".
Let us assume the "X" and "Y" actors each annually had to implement 10 activities. "X" actor has implemented three activities, has started implementation of two activities, and hasn't started at all implementation of 5 activities. When we apply the evaluation mechanism described above the "X" actor's performance equals 40:
((3*1) + (2*0.5) + (5*0)/10)*100 = 40
Whereas "Y" actor has fully implemented 2 activities, has started implementing 4 activities, and has not at all started implementing 4 activities. As a result, "Y" actor's performance equals 40 as well:
((2*1) + (4*0.5) + (4*0)/10)*100 = 40
As such, the performance indicator of these two actors is the same based on the formula as mentioned above. However, this approach does not take into account the workload of the actors that are tasked with implementing the activities indicated in the action plan of the Strategic Road Map. The resources and time commitment expected from implementing actors can differ. Meaning that the time commitment and intellectual labor required from the "X" actor that has fully implemented three activities can overpower the time and intellectual labor that has been committed by the "Y" actor which has fully implemented 2 activities and started implementation of 4 activities. Moreover, this is also important to understand what the actors mean by "starting the implementation of an action." At times some unproductive working group meeting on a certain objective or action can be presented as "starting the implementation of an action."
2) Solution to the problem mentioned earlier is differentiation of the activities by the categories such as technical and transformative. Implementation coefficient of some activities can be applied on a differentiated basis from other activities with a different system. Another hurdle for monitoring and evaluation is either the absence of outcome indicators of the activities or generic articulation of some of the indicators (e.g., the provisions: "1.3.5. - Formulation and management of adequate economic expectations" and "4.1.1. - development of free competition environment").
3) According to this methodology, "the specific weight of the activities in the overall scheme of activities the actors are responsible for implementing is not taken into account while evaluating the performance of actors." Therefore, the actors that have the least amount of activity responsibility (1 in total) in performance list come as leaders (Council of Financial Stability and Commission). If the differentiated coefficients mentioned above were applied, then the list of responsibilities of the actors can be taken into account as a separate variable.
4) The methodology of gathering data and sources of data must be indicated in the reports while assessing the activities implementation of which started by the executive power in 2017. The main implementing organizations present the quarterly reports on the activities to coordination council, coordination council after the analysis and assessment present the reports with a list of recommendations to the main implementing actor. "According to the Strategic Road Map on Perspectives of National Economy while conducting monitoring and evaluation the implementation of the activities will be assessed based on compliance of the activities with the objectives, expected outcome and indicators, and compliance with the implementation period" (page 107). The adequate evaluation of the activities can be carried out not only based on the information received from implementing organizations but as well as based on the information received from the beneficiaries and stakeholders ("stakeholder approach").
5) Reflecting the problems in the implementation of the Strategic Road Map in a half-yearly report is a positive development (e.g., not enough work has been done on the attraction of foreign private investment, building the infrastructure of e-trade following international norms, and entrance to global markets with more favorable trade conditions). However, the half yearly and the annual monitoring and evaluation reports could also reflect and provide a detailed explanation of the activities the implementation of which the state actors have either not started or not fully implemented.
Besides compliance with the implementation period and relevance, efficient use of resources, focus on the outcome and sustainable impact ("3E" approach) should also be taken into account while implementing the activities. In conclusion, to sum up everything that is mentioned above, our main recommendations are: designing the monitoring and evaluation mechanisms of the Strategic Road Map based on transparent and methodologically sound rules; designing system of indicators; application of differentiated coefficients while assessing the activities by taking into account content of the activities and workload of the implementing actors; including the ideas of the beneficiaries of the implemented activities and the stakeholders by stakeholder approach into the monitoring and evaluation mechanism; and increasing the volume of the report and the content. The following sources reflect some of the international best practices we have used for preparing our recommendations:
a. Guidelines for Project and Program Evaluations, Austrian Development Cooperation
b. Guidelines for design, implementation, and evaluation of Russian Federation's National Programs