D1.1 - Current evaluation practices and related challenges

Current evaluation practices and related challenges of policy evaluation and bridging ex-post and ex-ante

The study underlines the importance of the economic evaluation of environmental and climate policies in the process of developing the policies themselves. This ensures that the costs of a policy are reasonable, ensuring that the public and decision makers are more likely to accept it.

The research suggests carrying out economic evaluations both before (ex ante evaluations) and after the implementation of a policy (ex post evaluations), to obtain more effective policies and interventions based on concrete data. Ex post evaluations can help improve the accuracy and reliability of ex ante evaluations, as they will be based on real-world data and experiences rather than assumptions and projections. While ex ante evaluations can inform the design and implementation of a policy and its evaluation principles, to improve its effectiveness and efficiency once implemented.

Linking the two types of assessments can be difficult due to a number of factors. One of these is for example the strong likelihood that policy making differs between the two types of evaluation, due to policy changes that occur during implementation. As a result, the two assessments may be based on different data sources and scales of analysis which are difficult to compare.

The report therefore provides a first attempt to outline multiple areas for improvement regarding the economic evaluation process, such as connecting experts and practitioners ex-ante and ex-post, promoting open access to data and improving the understanding of complex ex-ante models.

D1.2 - Best Practices in Economic Appraisal Methods

Best Practices in economic appraisal methods

The study provides an analysis of economic evaluation methods used in climate and biodiversity policies, aiming to extract the best practices that can be useful in the PATTERN project case studies. It consists of four sections: the first provides an overview of the main approaches related to the analysis proposed in the case studies; the second examines various applied case studies to extract common best practices; the third focuses on the proposed methods for evaluation, such as carbon handprint, Q-method, or life cycle impact assessment; the fourth illustrates the main challenges and needs identified in the analysis. Some of the identified good practices include: (i) a clear definition of the scope of the analysis; (ii) the use of appropriate discount rates; (iii) the use of reliable data and the assessment of all relevant tangible and intangible costs and benefits; (iv) sensitivity analysis to understand how key assumptions change or how variables can influence results; (v) consultation with relevant professionals to ensure that the analysis reflects their concerns and interests.

Overall, the best practice for using economic evaluation methods is to use a combination of approaches and carefully consider the context and specific objectives of the project or policy.

D2.2 - Building a Theory of Change to identify evaluation needs

Building a Theory of Change to identify evaluation needs

Explore PATTERN’s innovative approach to policy evaluation through our latest document on the participatory crafting of Theories of Change (ToCs). This comprehensive endeavor involves stakeholders, policymakers, practitioners, and evaluators, forming a crucial bridge between ex-ante assessments and ex-post evaluations within our diverse Case Studies.

PATTERN’s strategic amalgamation of Utilization-Focused and Realist Evaluation sets the stage for a dynamic understanding of prospective and retrospective policies. Delve into the study to discover the intricate processes involved in policy development, from conceptualization and planning to intervention and retrospective evaluation.

This document not only sheds light on PATTERN’s unique methodology, but also invites you to join the ongoing conversation on rational planning, informed decision-making, and the pivotal role of evaluation in refining policies over their lifecycles. Immerse yourself in the intellectual journey of incremental circularity, where knowledge continually evolves, and evaluation becomes the compass for navigating unforeseeables in policy development.

D2.3 - Social Side of Policy Impact

Assessing the social side of policy impact

PATTERN’s methodological approach extracts a Theory of Change from stakeholder assumptions and introduces the Accountability Ceiling to distinguish direct policy impacts from broader influences. The framework categorizes social impacts into six domains: Capability, Community, Inclusiveness, Labor, Political Participation and Civil Rights, and Way of Life, each with dimensions assessed through specific indicators for thorough evaluation.

The document acknowledges challenges in directly attributing social impacts to policies due to the complex interplay of factors and proposes innovative approaches. The conclusions endorse this integrated, stakeholder-driven approach, urging policymakers and practitioners to adopt the framework to ensure effective and equitable climate and environmental policies.

D4.3 - Guidelines to bridge ex-post and ex-ante for policymakers

Aligned evaluations for more evidence-based and effective policy interventions.

Economic assessments of environmental and climate policies are crucial for effective policy-making.
Evaluations ensure costs are reasonable in relation to benefits, increasing public and decision-maker acceptance. Ex-ante appraisals compare policy options before implementation, while ex-post evaluations provide valuable lessons learnt for future decisions. Aligning these evaluations could lead to more evidence-based and effective policy interventions. Ex-post evaluations can help to improve the accuracy and reliability of ex-ante appraisals, as they will be based on real-world data and experiences rather than assumptions and projections. On the other hand, ex-ante evaluations can inform the design and implementation of a policy, and its ex-post evaluation, to improve its effectiveness and efficiency once it has been implemented (PATTERN D1.1, 2023).
A meta-analysis of current practices indicates a very limited use of ex-ante appraisals by ex-post evaluations or vice versa. This limited application is due to multiple challenges and uncertainties
happening throughout the evaluation steps, ranging from the intervention logic to the dissemination of the evaluation findings (PATTERN D1.1, 2023). Therefore, this report provides multiple ways of bridging both types of appraisal methods covering the policy evaluation process, giving stepwise and illustrated guidance on how ex-post analysis can improve ex-ante assessments, and vice versa, while conducting the actual appraisal.
More specifically, the guidance is covering two main parts, with Part A focusing on building possible bridges per evaluation step of economic appraisal, while Part B is a guide to bridging specifically for PATTERN’s economic appraisal methods.

 

D4.4 - Participatory approaches to Evaluation

Refining Theories of Change in Policy Design

This report presents the findings of Task 4.3, conducted by UNITO and the PATTERN Consortium, which focuses on enhancing the Theories of Change (ToCs) initially developed in WP2. Task 4.3 builds on the project’s interdisciplinary and participatory approach to evaluating climate and biodiversity policies, incorporating insights from WPs 3 and 4 to assess social impacts more comprehensively. The optimized and comprehensive ToCs for each case study are included in this report.
In previous work, Task 2.2 engaged practitioners and academics to derive a Theory of Change based on the assumptions and rationales of key actors (Rogers 2014; De Silva et al. 2015; Anderson 2017), while Task 2.3 examined the social dimensions of policy impacts, focusing on how these policies affect social relations and community well-being. Building on these foundations, this report emphasizes the importance of an interdisciplinary participatory approach in policy evaluation, presenting revised ToCs that incorporate insights from ex-ante and ex-post analysis detailed in D4.2 and D3.2.
As part of our participatory approach, described in D4.4, we worked closely with case study leaders and partners to evaluate the feasibility and evidence basis of the updated ToCs. Individual meetings with country leaders allowed us to assess the practical effectiveness of the revised ToCs, ensuring they reflect real-world conditions and are adaptable to each context.