Unlocking High-Quality Teaching

This annex presents the methodological approach followed in this report carried out under the OECD Schools+ Network initiative. By placing schools at the centre, this initiative seeks to draw insights from the frontline of our educational systems to improve education policy and practice, and provide a space to build bridges across policy, practice, and research. The methodology is one that, reflecting the objective to build bridges, has been characterised by a multi-stakeholder, collaborative and iterative development.

The overall aim of the work has been to to advance the understanding of school-level practices. To this end, the work has focused on three concrete goals:

• Goal 1: Develop a taxonomy of teaching that cuts across different pedagogies and provides shared language to facilitate a multi-stakeholder dialogue and knowledge exchange.

• Goal 2: Identify the best research evidence available for the practices included in the taxonomy and provide an indication of the respective areas of strength and limitation.

• Goal 3: Leverage the professional knowledge of schools on the implementation of the practices included in the taxonomy to further understand their complexity.

The Schools+ Network collected a range of different data to achieve these goals. These included the following activities:

• Meetings of the Informal Expert Group to develop background documents on the taxonomy of teaching

• Rating exercise with experts and organisations on the strength of evidence of practices, supported by qualitative insights on specific strengths and limitations

• Online surveys and meetings with schools to derive qualitative insights on challenges and approaches to implementing teaching practices

• Rating exercise with schools on the complexity of different teaching practices

• Consultation with experts, organisations and schools on the terminology and conceptualisation of teaching practices.

The following types of data were collected:

• School questionnaire on their background, practices and attitudes towards research and teaching practices

• School questionnaire on the terminology used to refer to practice and open qualitative comments on background documents

• School questionnaire on teaching decisions and signals from students

• Expert ratings on the strength of the best evidence available from evidence brokerage organisations and academics, supported by open questions on specific strengths and limitations

• Targeted comments on specific areas of the chapter and opportunities to openly review them from schools

• Expert ratings on the complexity of practices from schools

• Open case study submission and peer review from schools.

As mentioned, the work of Schools+ has depended upon the contributions of a range of different stakeholders. The Schools+ Network consists of two broad groups: Network participants and participating schools.

Since its launch on 22-23 May 2023, the Schools+ Network has grown to include over 50 participating institutions, such as ministries of education, local authorities, teacher and school leader organisations, large school networks, evidence brokerage organisations, and entities supporting educational development like philanthropic foundations and international organisations (see Table A A.1 for a complete list). Participants are invited to participate in two Global Community meetings annually, held in-person and virtually, to provide feedback on ongoing work.

To better capture schools’ practices and expertise, participants in the Schools+ Network nominated schools to join the ‘Learning Circle’. Schools were nominated based on their experience with research evidence and their interest in innovation. Over 150 schools from 40 countries, representing around 140 000 students, convened quarterly to further enhance our understanding of teaching practices. More information on the composition of schools can be seen below (see Figure A A.1). Schools were not designed to be a representative sample, and this should be considered when interpreting insights from schools.

The work of the Network has also been supported by two informal groups. First, the Informal Advisory Group. This group has provided ongoing support on coordination aspects of the Network with representatives from two OECD member countries (France and Portugal), five global and regional organisations (Education International, European School Head’s Association, International Conference of Principals, SUMMA, and UNESCO), and two philanthropic organisations (Jacobs Foundation and the LEGO Foundation).

Second, the Informal Expert Group. Consisting of academics from different institutions, this group has supported the development of the Pedagogical Taxonomy, reviewed the current research evidence, and provided advice on the expertise to be sought from schools (see Table B.3 in Annex B).

The Network has also relied on the contributions of numerous experts who have taken part in consultations. Table B.5 in Annex B provides a list of the 26 experts from academia and knowledge brokerage organisations that contributed to the rating exercise and review of scientific evidence on practices. Additionally, a wider group of academics and organisations also contributed qualitative input on the conceptualisation of practices and the scoping of their evidence base (see Table B.5 in Annex B).

The Schools+ Taxonomy of Teaching is the backbone of the work on advancing the understanding of teaching practices. It provides a “shared language”, a framework for educators, researchers, and policymakers to discuss and better understand classroom practices.

An iterative, inclusive, and participatory approach to the development of the Taxonomy was considered paramount to ensure its terminology and descriptions were appropriate and shared across different stakeholders. The main milestones are noted below.

Together with the OECD Secretariat, the Informal Expert Group (see Annex B) developed a preliminary version of the Taxonomy. A major reference was the OECD’s Global Teaching InSights Observation System which was developed for a pioneering Video Study that drew upon multiple methods to observe and document teaching in a detailed way, whilst also investigating which aspects of teaching are related to student learning and student non-cognitive outcomes.

The Global Teaching InSights Observation System was developed through four cycles of extensive cross-country collaboration between observation and pedagogical experts between 2015 and 2020. This observation system undertook the challenge of designing measures of teaching practices that would be applicable, valid and comparable across countries and across a variety of cultural contexts. To achieve this, the development drew upon four importance sources: participating countries’/economies’ conceptualisations of teaching quality, a review of relevant international research literature on the topic, and the conceptual frameworks of both the OECD’s Teaching and Learning International Survey (TALIS) and the Programme for International Student Assessment (PISA).

The initial phase of work on the Schools+ Taxonomy focused on scoping which practices could be carried forward from that reference framework and which other frameworks should be examined in more detail. It resulted in the identification of five major goals of high-quality teaching and an initial list of potential practices to be further investigated. Table A A.2 presents a summary of the differences between the Schools+ Taxonomy and Global Teaching InSights Video Study Observation System.

Each of the broad five teaching goals of the proposed Taxonomy was assigned a lead author from the Informal Expert Group, who refined a list of practices and examined the evidence behind each practice, with particular attention to empirical studies, meta-analyses and systematic reviews. Each author also provided an initial descriptor to define each practice. The proposed practices and definitions from across the five goals were collectively further refined through both online and in-person meetings, leading to a first version of the Taxonomy. This version included 22 practices organised into the five goals, complete with definitions and associated terminologies, supported by background documents that provided the conceptual understanding for each goal.

The OECD invited more than 100 experts from academia and knowledge brokerage organisations to partake in an expert review process. A total of 43 leading academics in the field of pedagogy, as well as education knowledge brokerage organisations, detailed in Annex B, provided feedback on the first version of the Taxonomy. The main goals of the review were to check on the alignment of the Taxonomy to different bodies of literature and ensure the appropriateness of definitions, whilst also developing an indication of the strength of evidence for each practice. Experts provided feedback independently from each other to reduce potential bias, and experts were invited to provide feedback based on where their expertise best aligned to the Taxonomy. This feedback was aggregated by the OECD, who then organised a series of discussion meetings with different stakeholders to further support the refinement of the Taxonomy, including an in-person meeting with a group of 14 knowledge brokerage organisations on 30-31 October 2023.

As the Taxonomy aimed to move towards a more shared language of pedagogy, a range of different education stakeholders were invited to comment on the first version of the Taxonomy. In particular, participating countries and organisations in the Schools+ Network were invited to provide feedback as well as over 150 schools from 40 different countries (see Annex B). The feedback sought was focused on the definitions and associated terms of the proposed core practices, as well as the conceptual background documents.

The consultation processes yielded valuable feedback that led to significant adjustments in the final version of the Taxonomy, such as the addition of Metacognition under Cognitive Engagement, the redefinition of Creating a Supportive Classroom Climate in the Social-Emotional Support dimension, and the relocation and merger of Crafting Explanations and Expositions with Explicit Procedures and Methods into the Quality Subject Content dimension (see Table A A.3). The feedback received also helped to provide greater precision and clarity to numerous descriptors, as well as yielding additional associated terms for each practice.

With an initial draft of the Taxonomy developed, the OECD Secretariat conducted a consultation process with experts and commissioned an independent literature review to identify the best evidence available and existing limitations of the evidence base.

Experts consisted of those working in academia and in knowledge brokerage efforts. A full list is provided in Annex B. Experts from academia were selected through a multi-faceted approach: (a) recommendations from the Informal Expert Group, to ensure alignment with the project's objectives; (b) from key references in the bibliographies of the background documents to include a breadth of disciplinary perspectives; and (c) nominations from participating experts to include additional scholars with valuable insights.

Objectives of the review

Experts were asked about the respective strengths and limitations of the evidence bases behind different practices included in the Taxonomy. Specifically, experts were invited to:

• Rate the strength of evidence for teaching practices using a shared set of criteria.

• Provide qualitative insights detailing the rationale behind their ratings, including any perceived limitations or strengths within the evidence base.

• Suggest key studies or ongoing debates that potentially merited inclusion.

• Share feedback on the definitions and terminologies used in the Schools+ Taxonomy to refine the conceptualisation of the practices (see Expert review process (September 2023 – November 2023) above).

• Provide qualitative insights on a set of key considerations that contribute to effective implementation in a classroom and the enabling conditions or barriers that schools may encounter when implementing certain practices effectively.

• Rate, based on their experience, how often practices were implemented in an effective, impactful manner in classrooms and to share through open text comments the challenges they felt teachers grappled with when trying to implement or refine the teaching practices. These were used to refine the key considerations that broke individual teaching practices in a more granular way.

Experts were invited to provide input only in areas of their expertise. This meant that experts could choose to provide feedback on some particular practices based on their expertise, on several or all of them. A total of 26 experts provided ratings on the strength of the evidence, though practices received a variable number of ratings. An additional 17 academics and organisations provided qualitative input on the conceptualisation of practices and the scoping of their evidence.

Experts provided their ratings independently from each other to help reduce potential bias from social conformity or from dominant individuals. Experts were given the same instructions on the process and an Excel document to conduct their ratings and share their comments through, that was then returned via email. To support consistent interpretation, experts were provided with the same definitions for each practice of the Taxonomy and the background work.

Processing experts’ review

The expert feedback was consolidated into a comprehensive master file, which recorded all the ratings assigned to each practice, along with verbatim transcriptions of experts’ qualitative comments. Basic statistics on their distribution and mean of ratings were calculated. The rationales accompanying the ratings were carefully reviewed, and key strengths and limitations were distilled into thematic categories.

One particular concern was ensuring consistency in the interpretation of evidence strength across reviewers. To address this, a series of steps were undertaken:

• Raw ratings of the evidence strength, aggregated ratings, and accompanying rationales were shared with the Chair of the Informal Expert Group to independently review (see Table A A.4).

• The Chair independently assigned an overall rating to each practice based on their analysis of the rating variation, the underlying rationales of ratings, and the detailed scoping of the evidence for each practice in the Informal Expert Group’s background documents (see Developing shared definitions (November 2022 – September 2023)).

• To ensure the objectivity and reliability of the findings, the OECD Secretariat conducted a parallel blind review too, replicating simultaneously the same process of assigning an overall rating as the Lead Expert.

• Any disagreements were reconciled through iterative deliberations and practices were organised into three groups that would reflect the degree of consensus on the causal impact of practices.

• For additional input, initial ratings were also reviewed and discussed with the wider Informal Expert Group and participating education evidence brokerage organisations at the aforementioned “Meeting on Linking Evidence and Practice in Education” to explore potential patterns and avenues that could be pursued.

Based on the final ratings assigned by the Chair and the OECD Secretariat, and reconsidering the distribution of initial ratings by experts, practices were systematically categorised into three evidence levels – high, medium, or low – reflecting the level of consensus on their positive causal impact on student outcomes. These categories were based on the number and quality of the studies, how consistent the findings were, and how well the evidence applied to different educational contexts, subjects, and levels. Alongside this, for each practice, the limitations in the evidence base for each practice were collated and synthesised. This involved the limitations identified by the Informal Expert Group’s background documents as well as the limitations identified in the qualitative insights by experts in the consultation process.

A literature review was commissioned to two independent reviewers in order to further examine the evidence available on the practices included in the Taxonomy. One purpose of the review was to check for bias and increase coverage in the evidence drawn upon by the Informal Expert Group in the background documents. This would allow a larger pool of evidence to be used in developing the final report. A second purpose was to also check for potential bias or oversight in the groupings of practices based on the expert consultation.

This literature review was designed following established evidence synthesis methodologies, ensuring that the breadth and depth of the analysed literature met the standards of reliability and validity. In particular, the protocol considered the following explicit inclusion and exclusion criteria:

• Type of Studies: The highest possible standard of evidence was used; when available, quantitative meta-analytic syntheses of randomised control trials or quasi-experimental designs.

• Population: Studies focusing on teachers in mainstream school settings in primary and secondary education levels (ISCED 1 to 3).

• Intervention: Inclusion of teaching practices, methodologies, and pedagogical approaches relevant to a specific sub-dimension.

• Outcomes: Measurable effects on student cognitive and non-cognitive performance as defined by the OECD Education 2030 Compass.

• Time Frame: Studies published post-2000, and, where a large meta-analysis exists for a sub-dimension, since the meta-analysis’ publication.

• Geographies and Language: Studies published in English or with available English translations.

The data sources and search strategy involved using keywords related to the sub-dimension and its associated terms to search databases such as ERIC (Education Resources Information Center) and Scopus. A data collection form was used to standardise the capture of study design features, methodology, findings, limitations, and conceptual alignment to the Taxonomy.

This meant that, overall, some 500 references have been considered (with some duplication) across the teaching goals of the Taxonomy: Classroom Interaction (95), Cognitive Engagement (139), Formative Assessment and Feedback (69), Quality of Subject Matter (88), Social-Emotional Support (171). It is also important to note that this distribution is informed by the strength of conceptualisations in different goals and also the nature of different evidence bases. For instance, high-quality meta-analyses are available for Formative Assessment and Feedback whilst similar references were missing for Cognitive Engagement and Social-Emotional Support. This is also reflected in the outcomes of the expert review process.

A further feature of the work of the Schools+ Network has been to gather and systematise teachers’ and school leaders’ professional knowledge through the Schools Learning Circle. From November 2023 to November 2024, over 150 schools have contributed to the Schools+ work.

Schools were nominated by participating organisations and ministries of the Schools+ Network. Each organisation or ministry were invited to nominate up to 10 schools. To facilitate the selection of these schools, a detailed guideline with criteria for the selection of schools was provided, this included the following criteria:

• Primary and/or secondary schools from different geographical areas and different socio-economic groups

• Be among the most pioneering establishments in their respective networks, demonstrating high standards of teaching and learning

• Make regular use of research evidence to make decisions and drive their school forward (e.g. working with researchers, engaging in action research, staying abreast of research findings, having strong school self-evaluation processes)

• Engage in innovation and show interest in how to inspire and support other schools to take initiatives to scale

• Passionate about shaping global education, sharing their expertise and co-creating solutions with peers from other countries.

A key expectation was that participating schools were attuned to research and making research-informed decisions that were supported by local evidence from their school context. Schools were recruited by completing an application form where they were asked to share information about the school’s characteristics, experience, and initiatives in relation to research and data use. When asked about it in the registration, more than 100 reported having participated in research projects, and nearly all had participated in competitions and local networks. Many schools also reported that they made use of different mediums for self-evaluation of in-school initiatives (e.g. assessment and student data, teacher questionnaires or testimonials). Schools were asked to include at least one member of the school leadership team and one member of the teaching staff, and were encouraged to form an in-school team of at least three staff members to allow for discussion and the development of synthesised school answers, with the following guidance for teams.

The majority of schools in the Learning Circle were public schools, with a reasonably equal split between those that catered to the primary, secondary or both primary and secondary levels (see Table A A.5). A notable proportion had a significant number of disadvantaged students based on their own estimates. The mean number of students per school was 934, and the mean number of teachers 88. Overall, it means that the Learning Circle has worked with schools that encompass more than 130 000 students and 12 000 teachers.

There is also a notable geographic spread in the participating schools as shown in Table A A.6. A total of 84 schools were nominated by governments (national and local) from OECD member and non-member countries and 67 schools were nominated by other Schools+ participating organisations.

Schools were organised into five thematic groups based on the five goals of teaching. Schools were asked to choose which of the five thematic groups/goals they were interested in focusing most of their attention on, receiving an overview of the five goals to inform their decision. Schools were encouraged to base their decision on their own practice and which thematic group/dimension they felt was a strength in their school such that they could share and build knowledge on its practices. This would also facilitate more focused reflections and discussions among participants by inviting them to concentrate on a single (if still broad) area of practice rather than teaching as a whole.

Each thematic group was also assigned two ‘co-leads’. These were schools who volunteered to undertake this role and expressed an interest in having a larger, more hands-on participation. From the group of interested schools, based on reported experience in working internationally and on specific evidence-informed initiatives in their school, as well as with consideration to geographic spread, a group of 10 co-leads were invited to undertake this important role.

Eliciting schools’ insights depended on a series of ‘Milestones’ that were shared with schools. Schools had a range of different stimuli and different outputs expected from them. Each milestone is outlined here.

Milestone 1 (November 2023) – Building a shared language of teaching and delineating the implementation mechanisms:

• To ensure a common understanding of practices, schools read the background document for their thematic area.

• Schools completed an online survey to share the language and terminology they use to describe practices, as well as their suggestions for how practices were described in terms of implementation mechanisms to inform the early development of insights on these.

• Schools met online to discuss themes across the submissions in their online surveys.

Milestone 2 (December – January 2024) – Identifying in-class signals from students and deepening implementation insights:

• Building on the background chapters still, schools were asked again through an online survey to share the types of in-class signals from students that they attended to when implementing each practice, both in terms of signals that a practice was and was not being effectively implemented.

• To build a richer set of insights on implementation mechanisms, schools were also asked to share how their implementation of different mechanisms in practices varied for different classes and contextual needs.

• Led by the co-leads, schools met online to discuss areas of disagreement in terms of mechanisms of effective implementation and the evolving synthesis of their insights.

Milestone 3 (February – April 2024) – Discussion on the implementation of practices:

• Schools had access to a shared ‘live’ version of the first draft of the chapters for the report for co-editing.

• A series of question boxes were placed at areas where, based on the previous milestones and discussions, there was still a lack of sufficient professional insights. Schools were invited to either comment on the existing text, share their insights through these question boxes, or engage with the ideas of other schools that had already been shared.

• In particular, schools were asked to share tangible examples of how they respond to particular challenges when implementing the practices to provide more meaningful insights on implementation mechanisms.

• Led by the co-leads, schools met online to discuss and refine the first synthesis of the signals from students based on Milestone 2.

Milestone 4 (June – September 2024) – Scoping the process of change and the enabling contextual factors:

• Schools were invited to share a detailed outline of how they have enacted a substantial change in their school to improve their use of at least one of the 20 practices.

• Schools shared their inspiring practices through a standardised template that had been developed through contributions from the Schools+ Third Community Meeting. This included schools sharing their school-level evidence on its impact and how school leaders had navigated different contextual factors at the school-level to ensure its effectiveness.

• Schools met online to discuss one case study example from a school in detail, and how to ensure that information on contextual factors would be detailed and relevant.

Milestone 5 (October – November 2024) – Understanding the fuller complexity of teaching:

• Schools undertook an expert rating exercise on the complexity of teaching. Schools rated each of the 20 practices for their inherent difficulty and the influence on contextual factors on their implementation. 1 This was completed in an online survey and more information on the rating exercise and the methodology to its processing can be found below.

• Schools also submitted qualitative insights on the types of effective instructional leadership practices that can help create a supportive environment for high-quality teaching.

• Schools met online to discuss in more detail the types of effective instructional leadership practices that respond to different contextual factors.

Milestone 6 (December – March 2024) – Exploring instructional leadership for change:

• Schools were provided with a background document on strategies to support teachers to refine their skills and create supportive environments (the initial draft of Chapter 8) in a ‘live document’ format, including the tentative results from the above rating exercise (Milestone 5).

• Schools were asked to share their feedback on how different contextual factors shape high-quality teaching and provide further insights from their own experiences of navigating these factors.

• Schools were given access to peer review the inspiring practices of their peers. In particular, the review focused on eliciting more detailed insights around how schools have gone through processes of change to shift behaviours and mindsets around practices.

As outlined, two particular types of insights were synthesised for inclusion in the report: insights on responding to key considerations when implementing practices, and real-time student signals that could be used to monitor the effectiveness of implementation. The development and refinement of these insights consisted of the following features:

• Triangulation of different data sources: Thematic analysis for the insights was conducted in an iterative way as additional school-level data was collected. Initial codes from Milestone 1 were steadily refined in a multi-step way, incorporating data from additional milestones.

• Practice-by-practice analysis: A decision was taken to use individual practices as the primary unit of analysis; each group had different practices that had their own set of particular implementation considerations and signals, both of which could be highly variable.

• Ongoing opportunities for collective refinement: There were opportunities for co-leads to review emerging themes from the data analysis, and space for discussion among schools in their thematic group meetings to consider themes and certain areas lacking clarity (e.g. particular signals, certain responses to key considerations).

• Monitoring participation to ensure breadth of expertise: Participation rates across milestones were attended to ensure a diversity of perspectives were being heard from.

• Links to other OECD frameworks: Student signals were mapped to the Education2030 Learning Compass to provide greater coherence when interpreting and discussing them.

• Additional checks: After synthesising insights and signals, original data was returned to schools for an additional check and shared with a broader number of schools.

As part of Milestone 5, schools were asked to complete two ratings for each of the 20 practices drawing upon their expertise:

• “What is the level of difficulty for an expert teacher to execute this practice, regardless of contextual factors?”

• “What is the level of influence of contextual factors (external to the expert teacher) on this practice?”

Schools were given guidance on how to approach the ratings2 and given an example of the type of rating page they would be using accompanied by an example of how to rate with both scales a non-classroom example3. Even if participating schools were already expected to be familiar with the definitions for each practice, the definitions were included for clarity.

Practices were presented in a random order to improve reliability. A 7-point Likert scale was adjudged to be best for capturing more variability in ratings, with 1 representing a very low difficulty or very low influence of contextual factors, and 7 a very high difficulty or very high influence of contextual factors. A question was included that gave raters the opportunity to identify up to five practices on each rating scale that were particularly hard for them to judge (see Table A A.7 and Table A A.8).

The rating instrument was first piloted with 32 schools. Then, all participating schools were then invited to undertake the ratings. 132 school leaders and teachers, from 87 different schools, completed the rating. There were 53 classroom teachers with the rest having school leadership roles. Raters had an average experience of over 19 years working as a teacher. Those who were principals or school leaders had approximately 10 years of experience in leadership roles.

To process ratings, the decision was taken to organise practices into groups. There was little conceptual value in the specific scores that practices received. It was more meaningful to identify the patterns that captured the relative differences between practices, rather than trying, for example, to quantify specifically how much harder a practice was than another. Analysis of the intra-class correlation suggested reasonable levels of consensus on which practices were more or less difficult, and more or less influenced by contextual factors. There was more limited agreement on the exact difficulty of or contextual influence on a particular practice (e.g. was it a 4 or a 3 on the 7-point Likert scale), which was to be expected on such a scale.

The boundaries of the groups were developed based on the mean average rating, cross-checked with the median and modal ratings. This cross-checking as well as the visualisation of the dataset and its distribution of ratings allowed for appropriate boundaries to be set, even if these boundaries are not fixed and practices can vary in their difficulty or influence by contextual factors. The final groups were as follows, where x represents the mean average rating across raters:

• Difficulty: Lower difficulty x < 3.2, medium difficulty 3.2 ≤ x < 3.7, higher difficulty 3.7 ≤ x.

• Contextual factors: Lower contextual influence x < 4.0, medium contextual influence 4.0 ≤ x < 4.5, higher contextual influence 4.5 ≤ x.

The Schools+ Secretariat requested that schools, under the direction of the 'co-leads' engage in Milestone work and attend regular meetings to report on their progress. The participation of the 150 Schools in the Learning Circle has been consistent, both in terms of Milestone assignments and meeting attendance. This is captured by the rates of participation in the first and last milestones, which show some drop off but reasonably sustained levels of engagement from participating schools. Milestone 1 saw some 111 schools participate whilst Milestone 5’s activity, the aforementioned expert rating exercise, saw a total of 132 ratings by school leaders and teachers from 87 schools.

A similar picture emerged in terms of participation in meetings. Meeting participation includes representatives from schools, along with other colleagues from the schools. All of the meetings took place on Zoom and recordings, presentations and other materials were shared with all participants following the meeting, to account for schools that were unable to attend. The meeting attendance is shown below: