Applying Evaluation Criteria Thoughtfully

A criterion is a standard or principle used in evaluation as the basis for evaluative judgement.

Each of the six criteria is summarised by a broad question, which illustrates its overall meaning. Each one represents an important element for consideration:

• Relevance: Is the intervention1 doing the right things?

• Coherence: How well does the intervention fit?

• Effectiveness: Is the intervention achieving its objectives?

• Efficiency: How well are resources being used?

• Impact: What difference does the intervention make?

• Sustainability: Will the benefits last?

The evaluation criteria’s purpose is to support consistent, high-quality evaluation within a common framework. They provide a normative framework with which to assess a specific intervention. The criteria can also be used in processes beyond evaluation, including defining frameworks and indicators for monitoring and results management, funding approval, strategic planning and intervention design, particularly to improve future interventions. Collectively, there is value in having commonly defined criteria that are similarly applied across interventions. The criteria also provide a consistent language across the development field, providing standardisation and allowing for comparison and learning across interventions.

The criteria should be viewed as a set of lenses through which one can understand and analyse an intervention. The criteria provide complementary perspectives, giving a holistic picture of the intervention. They encourage deeper thinking about the nature of an intervention, its implementation, process and results. Together they describe the desired attributes of interventions, make explicit assumptions and provide norms: that interventions should be relevant to the context, coherent with other interventions, achieve results in an efficient way and have positive, lasting impacts for sustainable development. The evaluation criteria also provide a widely accepted framework for developing an approach to evaluation, a comprehensive and systematic approach and a common language that is used from the very start of the evaluation process.

The criteria are related and can be seen as a complementary set, to which each criterion brings a different and unique perspective to the understanding of the intervention. The definition of each criterion presents a distinct concept (captured in the overarching question for each) and yet these concepts are in many ways interrelated. Development interventions are typically multifaceted so using the criteria relationally can help the evaluator to consider the intervention as a whole.

The evaluation criteria are not a methodology and they are not the goals that an intervention is trying to achieve. Instead they provide prompts for asking the right questions during the evaluation of an intervention. Every intervention (the object or evaluand2) is different, which is why the evaluation process needs to be flexible and the use of evaluation criteria should be thoughtful and adapted to the purpose and users. In accordance with evaluation quality standards, conclusions and recommendations about progress and performance should be based on appropriate, credible evidence. The logic, credibility and interpretation of evidence should be clear and follow the theory of change or logic of the intervention.

The term “intervention” is used throughout the guidance to mean the topic or object of the evaluation. It encompasses all the different types of efforts that may be evaluated using these criteria. These can be either international or domestic, aiming to support sustainable development or humanitarian goals.

An intervention may denote a project, programme, policy, strategy, thematic area, technical assistance, policy advice, an institution, financing mechanism, instrument, or other activity. It includes development interventions, humanitarian assistance, peacebuilding support, normative work and other international co-operation activities, as well as the activities of private sector actors and national governments in domestic policy contexts. The criteria have been used, for example, to evaluate topics ranging from the efficiency of national school feeding programmes, to the coherence of international support provided by different actors in a conflict-affected region. They are also used to evaluate policies or strategies (e.g. see Box 2.1). Evaluations also use the criteria when evaluating a suite of interventions – for example an evaluation of various projects supporting the education sector or a single large “intervention” lasting many years and involving multiple partners (e.g. general budget support).

The term “intervention” is used in this document as it is the most easily understood, but evaluators should take care to define clearly the topic of the evaluation – the intervention being analysed – and the evaluation scope, early on in the process. Using other more specific words related to the intervention in question – such as project or policy – is likely to be more useful and more readily understood by partners.

Because every intervention (the entity being evaluated) is different, the evaluation process needs to be flexible and the use of evaluation criteria should be thoughtful and adapted to the purpose and users. The purpose of the criteria is not therefore to provide a mandatory set of rules.

The criteria do not exist in isolation. To make best use of the criteria it is important to understand where they fit in relation to other norms and standards, methodologies and institution-level guidance.

• The macro level is where the core principles for evaluation are situated, such as impartiality and independence, credibility, usefulness and participation (OECD, 1991[2]) along with the Quality Standards for Development Evaluation (OECD, 2010[3]). This level also includes the ethical standards related to research and data collection. These provide the overarching standards for good practice that all evaluations should adhere to throughout the process.

• The evaluation criteria are found at a meso level, providing the lenses through which the intervention is analysed and understood.

• The institutional level is where each organisation adapts the criteria and translates them into their own guidelines and policies (e.g. emphasising certain criteria, mandating ratings, adding additional criteria), reflecting their own mandates and priorities.

• The micro level, i.e. within the context of each individual evaluation, includes the decisions about the evaluation questions and methodologies that are employed for each evaluation, and how the criteria are applied in that context and to that specific evaluand.

[3] OECD (2010), Quality Standards for Development Evaluation, DAC Guidelines and Reference Series, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264083905-en.

[2] OECD (1991), Principles for Evaluation of Development Assistance, Development Assistance Committee, OECD Publishing, Paris, https://www.oecd.org/development/evaluation/2755284.pdf (accessed on 11 January 2021).

[1] Schwedersky, T., L. Ahrens and H. Steckhan (2017), Evaluation of the BMZ Action Plan for the Inclusion of Persons with Disabilities, DEval, https://d-nb.info/1186644206/34 (accessed on 11 January 2021).