Monitoring, evaluation, and learning in the Rio Doce Panel
A strategy to comprehensively assess the direct and indirect influence of the Panel's recommendations on the reparation of the Rio Doce Basin
Monitoring a project's activities is essential for keeping track of its execution, identifying the results achieved (as well as potential improvements) and keeping track of the ways in which the project contributes to accomplishing its mission. The Rio Doce Panel Project has a consolidated Monitoring, Evaluation and Learning (MEL) strategy that, in addition to establishing performance indicators and execution targets for all activities, also measures their effectiveness and captures elements that evidence the Panel's contribution to the reparation of the Rio Doce Basin.
What is the purpose of a monitoring, evaluation, and learning strategy?
Project operations must be supported by a rational framework that can describe how a given outcome will be achieved by means of the proposed activities. Though it may seem like a simple idea, the activity-result-impact sequence carries several assumptions and expectations on how a project should work.
Do the project staff perform their activities on time and deliver the expected quality? Are the planned activities able to achieve the expected outcomes? Do the outcomes lead to the expected change? Are there other stakeholders involved or other drivers outside the project that could influence its outcome or impact?
Employing indicators explicitly built for each project, it is possible to monitor the execution of planned activities (e.g., the preparation of a study) and their outcomes (e.g., the study's reach into, and acceptance by, the target audience). The impact of project outcomes on achieving goals and objectives (e.g., how the study influences public policy actions and decision-making) can also be monitored.
The purpose of ongoing monitoring is to identify instances where the project may not be operating as expected or achieving the desired outcomes, thereby providing an adaptive management opportunity. For example, if outputs and activities are not delivered on time, the reason for the delays should be investigated and addressed. If activities are delivered, but their outcomes are not achieved, the assumption that the activities are enough to achieve the outcomes can be revisited. Estimates should be reviewed if outcomes fail to produce concrete impacts; it is also good to check any external drivers that may have changed the context for which the objective was devised.
For this reason, the objective of the strategy is to provide a learning experience for all stakeholders and staffers involved by recording joint experiences and favouring institutional learning, with elements that can be applied to other projects in the future.
The Monitoring, Evaluation and Learning (MEL) strategy of The Rio Doce Panel
The starting point for the Rio Doce Panel's MEL strategy is the Theory of Change (ToC), as it describes how the project lays out the activities needed to generate the outcomes - that is, the way these outcomes lead to concrete impacts and how the impacts contribute to the Panel's long-term vision.
The ToC was built considering the map of the stakeholders involved in achieving the project's objectives. It also lays out the project's assumptions by describing how it intends to achieve its objectives.
The ToC describes the main elements that guide the establishment of indicators and the monitoring strategy itself, which include:
- Production and delivery of knowledge products (studies: Issue Papers and Thematic Reports) as per the annual work plans agreed upon by the Renova Foundation.
- Institutional support, communication and monitoring activities by the International Union for the Conservation of Nature (IUCN).
- Access to knowledge products by the target audience as mapped out by the Panel - for example, access-related data on the Panel's website, broken down by country; the number of times studies have been downloaded; mentions of the Panel's work in reports by organisations involved in the reparation work; minutes of public meetings; media articles; scientific articles, etc.
- Stakeholder agreement with, and support of, recommendations by means of interactions and direct feedback from the Renova Foundation and surveys conducted with other groups that interact with the Panel.
- Use of the recommendations, as identified through direct interactions and feedback from the Renova Foundation, as well as a review and analysis of secondary information, such as reports on the activities of different stakeholders, communications materials, and meeting minutes, among others.
In addition to monitoring whether the expected change takes place as expected, the MEL strategy leverages tools to evaluate any unexpected influences the work may have. Such influences can be recorded by anyone working on the project when they come across evidence that the Panel's work has influenced a stakeholder.
The monitoring of Project execution, outcomes, and impacts in accordance with the aforementioned points allows for strategic follow-up on the Project and the adaptive management framework when necessary. Ongoing Project monitoring makes it easier to prepare mandatory reports for various levels and serves as evidence of the efforts put in place.
Information collected using this strategy can tell the story of the Project's influence by depicting how the activities have led to results and, eventually, to achieving the Project's objectives.
What can the MEL strategy tell us about the Panel's results so far?
- The Renova Foundation has heeded and is implementing the Panel's recommendations to perform a comprehensive and systematic assessment of the impacts of the dam failure.
- The Panel's publications on impact assessment (Thematic Report 1 and Issue Paper 4) and climate (Thematic Report 2) influenced and were mentioned in the Recovery Plan for the Paraopeba River Basin, located in Brumadinho, which was hit by a similar disaster.
- The academic community uses the Panel's papers as a reference in the Fundão dam disaster: they have been mentioned 18 times in scientific journals since 2019.
- The Panel contributed to the discussions on developing a Global Standard for safety measures in tailings dams, with a particular focus on the need for post-disaster action plans.
- The Panel website is widely accessed by domestic and international visitors (between 10 and 12.6 page views per year). More than 12 thousand users have downloaded the Panel's outputs in Portuguese and English.
Please visit the link to view the MEL report for 2021 and previous years.