Do Adults Have the Skills They Need to Thrive in a Changing World?

The second cycle of the Survey of Adult Skills (a product of the Programme for the International Assessment of Adult Competencies – PIAAC) took place in 2022/2023. It was originally planned to take place in 2021/22, ten years after the first round of the first cycle of the survey. Because of the COVID-19 pandemic, all activities were postponed by one year.

The pandemic had significant, although difficult to quantify, impacts on individual and societal attitudes and behaviours. Attitudes towards participating in surveys changed after the pandemic, even more than what was observed in previous years. This is reflected in declining response rates for the 2023 Survey of Adult Skills compared with the first cycle of the survey, as well as for many other household surveys. Considerable efforts were made by the countries participating in the survey to maximise response rates and to minimise the effects of non-response.

The overall quality of the data and the adherence to the technical standards for the administration of the survey were evaluated by independent experts, members of the Technical Advisory Group (TAG) in May 2024. This evaluation involved a holistic assessment of the quality of the data, including: the representativity of the data with respect to the target population, the comparability of results across countries, and the adherence to the technical standards. The TAG concluded that all countries adhered satisfactorily to the PIAAC standards and that data from all countries and economies are of sufficiently high quality for publication and reporting.

Additionally, extensive analyses were conducted to evaluate the extent to which estimates from the survey are susceptible to non-response bias. A summary of the non-response bias analysis as well as a few country-specific issues that readers are invited to keep in mind while interpreting the Survey’s results, are presented in this Readers’ Guide. More information can be found in the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]).

In the first round of the Survey of Adult Skills, undertaken in 2011/12, response rates varied between 45% and 75% (OECD, 2024[2]). In 2023, response rates were much lower, varying between 27% and 73%. Korea was the only country where the response rate exceeded 70%.

Low response rates are a potential source of non-response bias. Bias from non-response can occur if non-respondents are systematically different (in terms of skills proficiency, for example) from those who agree to participate in the survey. For this reason, the 2023 Survey of Adult Skills undertook extensive non-response bias analyses (NRBAs). These NRBAs considered, for each participating country and economy, a range of indicators and analyses to gauge the likelihood of bias related to non-response in the estimation of adults’ proficiency from the survey. Details on the indicators included in the analysis and its final results can be found in the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]).

One of these analyses compared the weighted sample of participants in the Survey of Adult Skills to other sources to identify potential misalignments in the distribution of certain characteristics, such as age, gender, immigrant status, level of education, or employment status.

When the decision to participate in the survey is linked to specific characteristics of the sampled adults, the distribution of these characteristics among participants in the Survey of Adult Skills may not match the true distribution in the population or that observed in other surveys. However, as all surveys are affected by some non-response, it is not clear that other sources are necessarily more accurate than the Survey of Adult Skills. Misalignments between the distribution of certain characteristics of adults in the sample and in the population, as well as with other sources, have been observed in both cycles of the Survey of Adult Skills. As a result, the changes in the composition of the population observed over the two cycles of the Survey of Adult Skills may also not match exactly the changes observed in other sources.

Sampling weights can be applied to bring the composition of the sample closer to the known distribution of characteristics in the population. In the Survey of Adult Skills, countries and economies agreed with the Secretariat and the OECD contractors on which variables to use for weighting. For variables that were not used in weighting, some discrepancies may remain after weighting.

Table 1 provides an overview of some misalignments between the (weighted) PIAAC sample and alternative data sources. Discrepancies are listed if they are statistically significant and over 1.5 percentage points. Some of these misalignments can be explained by differences in the definition or in other methodological aspects between the Survey of Adult Skills and other sources. Moreover, not all countries could verify the alignment across all characteristics due to data availability.

Alternative weighting schemes were considered to understand the extent to which the observed discrepancies might bias the estimated proficiency of the adult population in the 2023 Survey of Adult Skills. Significant differences in estimated proficiency were found in a few countries with these alternative weights (Table 2). However, the impact of applying alternative weighting schemes never exceeded four score points. This means that the fact that the distribution of some characteristics in the sample is not consistent with other sources does not seem to have a major impact on estimated proficiency because these alternative weighting adjustments resulted in only minor changes to the overall results.

The final outcomes of the NRBA are reported in Table 3. These are based on the analyses described above but also on other indicators that are described in more detail in the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]). Countries were assigned to different categories, reflecting an assessment of the likelihood that their proficiency estimates could be affected by non-response bias and the extent to which results should be interpreted with caution. Such categories are:

• Pass, meaning that the analysis provided no strong evidence of non-response bias;

• Low caution, meaning that some caution should be taken in interpreting the results, as non-response bias may be present;

• Medium caution, meaning that proficiency estimates are more susceptible to non-response bias and more caution should be exerted;

• High caution, meaning that the likelihood of non-response bias is higher.

Table 3 also presents response rates for each country and economy. Response rates are only one element (although an important one) considered in the NRBAs, as mentioned previously. It is, therefore, possible that countries with lower response rates receive a more favourable assessment than countries who achieved higher response rates.

In interpreting the outcomes of the NRBA and the resulting classification of countries, readers should note that:

• The classification reflects an assessment of the likely existence of non-response bias, and not of its magnitude. In other words, one cannot conclude that countries in the “high caution” category necessarily have a larger non-response bias than countries in the “low caution” category.

• The criteria and the thresholds used for the classification have been approved by the independent experts who are members of the TAG.

• In any such classification, threshold effects will always exist, meaning that countries might be classified in different categories even though the differences in the underlying indicators are very small.

• This classification reflects a judgement on the collected data, and not on the quality of the work done by national centres and data collection agencies, which all countries completed satisfactorily and in accordance with the requirements specified in the Technical Standards.

In Lithuania and the Slovak Republic, not all eligible persons in a household were given a chance to be selected to participate in the survey, which could lead to coverage error. Measures were taken to reduce coverage error (weight calibration). While some additional caution should be taken when analysing data from these countries, the results of additional analysis, including the NRBA, suggest that the effect of this deviation from the sampling standards is rather small.

The OECD Secretariat conducted additional quality checks on the data, expanding on the quality control procedures set in the Technical Standards and Guidelines. These analyses found some unusual response patterns, suggesting that some respondents may not have exerted a reasonable level of effort in answering the literacy, numeracy and adaptive problem solving assessment. This may call into question whether their responses accurately reflect their proficiency. To identify such cases, the OECD relied on the following criteria: a very short time spent on the assessment, a high share of very rapid responses, a high share of missing answers, and locator failure (i.e. failure to answer a set of easy questions) from highly educated, native-born respondents. Anomalies were mostly found in the responses to the cognitive assessment; the pattern of responses to the background questionnaire did not raise particular concerns.

Disengaged respondents will always exist in surveys, and it is difficult to establish objective criteria to assess whether a reasonable level of effort was exerted. In some countries (Israel, Lithuania, New Zealand, Poland, the Slovak Republic, and Spain), it was found that many of these respondents were clustered around a few interviewers, suggesting that the problem may stem from such interviewers not following the PIAAC protocols. In particular, interviewers were identified for which a high share of their cases met at least two of the criteria mentioned above. This cast doubts on the quality of all data collected by such interviewers. The following notes describe in more detail the results of the analysis in these six countries, and the actions undertaken by the OECD to improve, to the extent possible, the quality of the published data.

Note regarding Lithuania, New Zealand, the Slovak Republic and Spain

In Lithuania, New Zealand, the Slovak Republic and Spain, all cases from the identified interviewers (406 in Lithuania, 301 in New Zealand, 356 in the Slovak Republic, and 385 in Spain) have been excluded from the data used to estimate the population model, which establishes the relationship between the variables from the background questionnaire and performance on the direct assessment to generate proficiency estimates (plausible values; see the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]) for more detail). This exclusion enhances the robustness of the model, by ensuring it is estimated based only on the cases considered to be of sufficient quality. In the absence of definitive evidence of data falsification or other forms of interviewer misconduct, the responses to the cognitive assessment items still contributed to the estimation of plausible values for these respondents.

Note regarding Israel

In Israel, six interviewers were identified as having a relatively large share of cases with unusual response patterns, using the same criteria that led to the identification of cases in the other countries. All data from these interviewers (748 cases in total) have been excluded from the data used to estimate the population model, which establishes the relationship between the variables from the background questionnaire and performance on the direct assessment to generate proficiency estimates (OECD, forthcoming[1]). This exclusion enhances the robustness of the model, by ensuring it is estimated based only on the cases considered to be of sufficient quality.

Moreover, stronger evidence was collected that three of these interviewers breached data collection protocols throughout the survey or were implausibly productive (conducting a very large number of interviews in a relatively short period). As this raised additional concerns about the quality of data, the responses to the cognitive assessment items for all cases of these interviewers were excluded from the database (572 in total). Plausible values for these cases were then estimated using only their responses to the background questionnaire (for which no unusual patterns were detected) and the parameters estimated by the population model.

Note regarding Poland

In Poland, nine interviewers were identified as having a relatively large share of cases with unusual response patterns of respondents, using the same criteria that led to the identification of cases in the other countries. All data from these interviewers (774 cases in total) have been excluded from the data used to estimate the population model, which establishes the relationship between the variables from the background questionnaire and performance on the direct assessment to generate proficiency estimates (OECD, forthcoming[1]). This exclusion enhances the robustness of the model, by ensuring it is estimated based only on the cases considered to be of sufficient quality.

Moreover, stronger evidence was collected that six of these interviewers in Poland breached data collection protocols throughout the survey. For instance, some of these interviewers were implausibly productive, conducting many interviews on a single day. Others did not record interviews or obtain respondents’ phone numbers, which made validation of interviews more difficult. Yet another interviewer was found to have falsified seven cases during data collection (cases which were immediately removed from the dataset as part of the quality control process and are not included in the 774 cases under consideration in this note). Twenty-seven other cases collected from this interviewer were, however, validated and remained in the dataset. Since these factors raise concerns about the quality of all cases completed by these six interviewers, the responses to the cognitive assessment items for all cases of these six interviewers were excluded from the database (559 in total). Plausible values for these cases were then estimated using only their responses to the background questionnaire (for which no unusual patterns were detected) and the parameters estimated by the population model.

In Poland, other cases with unusual response patterns that could suggest possible disengagement or lack of a reasonable level of effort during the assessment were identified. As these cases were not clustered within any particular interviewer, they were left in the dataset and treated as all other cases, given the difficulty of establishing objective criteria to determine whether reasonable effort was exerted, and whether the results of the assessment truly reflect the proficiency of respondents. While similar cases are present in all countries, the number of such cases in Poland can potentially have a significant impact on the estimated proficiency of the overall population. This should be kept in mind when interpreting Poland’s results.

The literacy and numeracy scores from the 2023 Survey of Adult Skills are psychometrically linked and, therefore, comparable with results from previous adult skills surveys, in particular the first cycle of the Survey of Adult Skills (conducted over three rounds between 2011/12 and 2017), the Adult Literacy and Life Skills Survey (ALL, conducted between 2003 and 2007) and (in literacy only) the International Adult Literacy Survey (IALS, conducted between 1994 and 1998). The data make it possible to analyse changes in skills proficiency over time. Survey operations also remained largely unchanged over time. All surveys were conducted by trained interviewers in the respondents’ households. The interviewer first administered a background questionnaire and then supervised participating adults during the direct assessment.

Some caution is always advised when comparing results across different surveys, especially as they were conducted many years apart and, therefore, under different conditions. Some methodological changes that occurred between the first and the second cycle of the survey need to be considered when analysing changes in proficiency over time. These are briefly summarised below. More information on the comparability of data across adult skills surveys can be found in the Survey of Adult Skills 2023 Reader’s Companion and the Survey of Adult Skills 2023 Technical Report (OECD, 2024[2]; OECD, forthcoming[1]).

In all countries, a small percentage of adults were unable to participate in the survey because they lacked the necessary language proficiency to communicate with the interviewer who administered the background questionnaire. For the same reason, they were also unable to participate in the cognitive assessment. These adults are referred to as “literacy-related non-respondents” (LRNR). In the first cycle of the Survey of Adult Skills, no information was collected on such adults, effectively leading to a small undercoverage of the target population and an upward bias in the average proficiency of the population.

In the 2023 Survey of Adult Skills, such adults were administered a very short questionnaire available in many languages (called “doorstep interview”). Respondents completed the questionnaire on a tablet by themselves. The information collected through this doorstep interview (age range, gender, years of completed education, employment status and migration history) was used to generate plausible values for these respondents, thus allowing them to contribute to the estimation of the average proficiency of the population.

The introduction of the doorstep interview creates a small misalignment between the populations for which proficiency estimates are available in the two cycles of the survey. For this reason, this report systematically excludes doorstep interview cases from the analysis when comparing results from the 2023 Survey of Adult Skills with those from previous adult skills surveys (including the first cycle of the Survey of Adult Skills). Doorstep interview cases may also be excluded because the information necessary for the analysis is not collected in the questionnaire. The inclusion or exclusion of doorstep interview cases is specified under every table and figure of this report.

The Technical Standards and Guidelines allowed for the background questionnaire to be completed with the aid of a translator or interpreter. This could have been a family member or a staff of the survey organisation. Sweden was the only country that made systematic use of interpreters in both cycles of the survey. As a result, all respondents in Sweden, in both cycles, were able to complete the background questionnaire, and no respondents were classified as literacy-related non-respondents (in the first cycle), nor took the doorstep interview (in the second cycle). This improved the precision of the estimates of proficiency for such respondents in Sweden because richer information is available for them. However, it also introduced a small difference in survey operations between Sweden and other countries, as adults who were not able to participate in the survey because of language barriers were treated slightly differently in Sweden than in other countries. As such adults constitute a very small percentage of the population, the threat to comparability remains low. However, some caution is warranted when comparing results between Sweden and other countries for groups of adults that may include a larger share of adults with very low language proficiency (for example, recent immigrants).

In both cycles of the Survey of Adult Skills, adults who failed the locator assessments were judged to have too low proficiency to undertake the full assessment. To gather some information on what such low-skilled adults are able to do, they were administered an assessment that tested the basic skills that are a prerequisite to developing proficiency in literacy (reading components). In the 2023 Survey of Adult Skills, a numeracy component assessment was also added.

Thanks to enhancements in the assessment design, the 2023 Survey of Adult Skills could incorporate the component assessment when estimating the proficiency of respondents. This increased the precision of proficiency estimates for those at the lower end of the skills distribution. This methodological change has a negligible impact on the estimation of average proficiency for the entire adult population. However, it has a more significant impact on the proficiency estimates of adults who failed the locator and only took the component assessment. In the first cycle, the proficiency of these respondents would have been computed differently, which would have led to sometimes significantly different results.

Results of all analyses that look at how the proficiency of subgroups of the population has changed between the first and the second cycle of the survey are not reported if, within the group analysed, the (weighted) share of adults in the second cycle who failed the locator and only took the component assessment is above 20%.

In the 2023 Survey of Adult Skills, the classification of levels of education is based on the latest International Standard Classification of Education (ISCED 2011), while the first cycle of the survey relied on the previous classification (ISCED 1997). These two classifications can be compared using official crosswalks.

However, over time, a few countries have changed the way they classify specific educational qualifications. These reclassifications have caused some qualifications to change ISCED levels. In this report, whenever results from the 2023 Survey of Adult Skills are compared with results from the first cycle, educational qualifications have been reclassified and assigned to the same level they would have been assigned in the first cycle (OECD, forthcoming[1]).

When examining changes in proficiency between the first and second cycle of the Survey of Adult Skills, additional uncertainty around scale values due to changes in the assessment frameworks and items must be taken into account (“is a score of 235 in the second cycle of the survey equivalent to 235 in the first cycle?”). The difference between a score in the second cycle scale and the corresponding score in the first cycle scale is modelled as a constant, equal for all countries and at all points of the proficiency scale. While the actual value of this constant remains unknown, its standard deviation ${le}_{\mathrm{1,2}}$ can be estimated and is known as the linking error. This linking error should be added to the standard error of any trend statistics expressed as a proficiency score (e.g. difference in mean proficiency across cycles or the values of the percentiles of the proficiency distribution). More formally, the standard error of the change in proficiency for country (or subgroup) $g$ between the first and second cycle is: $\sigma \left({∆}_{g2-g1}\right)=\sqrt{{\sigma }_{g2}^2+{\sigma }_{g1}^2+{le}_{\mathrm{1,2}}^2}$, where ${\sigma }_{g1}$is the standard error of the proficiency of group $g$ in the first cycle, ${\sigma }_{g2}$ is the standard error of the proficiency of group $g$ in the second cycle, and ${le}_{\mathrm{1,2}}$ is the linking error between the two cycles. The actual value of the linking error is 3.27 for literacy and 2.95 for numeracy.

It should be noted that the linking error does not apply to trends of any statistic which is homogenous to a score point difference, such as gender or age gaps in proficiency scores. Given that the additional uncertainty for comparing results across cycles is modelled as a constant at all points of the scale, when taking the difference between two scores, the uncertainties associated with each score cancel each other out, and there is no need to add the linking error term to the standard error of the trend.

A more complex case is when the analysis looks at trends in the shares of the population scoring at a given proficiency level. In this case, the additional error term for the standard error of these trends depends on ${le}_{\mathrm{1,2}}$, but also on the density $f_g$ of the proficiency score distribution around these cut-offs. For instance, the resulting linking error for the trend in the proportion of the population score at Level 1 in group $g$ will be ${le}_{\mathrm{1,2}}^2*({f_g\left(226\right)-f_g\left(176\right))}^2$.

The 2023 Survey of Adult Skills included a new assessment of adaptive problem solving (APS). This new domain replaced the assessment of problem solving in technology-rich environments (PSTRE) administered in the first cycle of the survey. As these two assessments do not share a common conceptual framework or any assessment items, the results from these two assessments are not comparable.

The following symbols are used to denote missing data:

Detailed data tables corresponding to the figures presented in the main body of the report can be found in Annex A.

These figures and tables share a common reference number, are numbered according to the corresponding chapters, and include an abbreviation in brackets to denote the skill domain they refer to (when applicable) – literacy (L), numeracy (N) and adaptive problem solving (A). As an example, Figure 3.1 denotes Figure 1 in Chapter 3; Table A.3.1 (L) is a table contained in Annex A, cited or referenced in Chapter 3, and containing data for literacy only.

Annex B includes other detailed data tables that either correspond to figures included in boxes or to citations in the main body of the report, but no figure was provided for them.

Unless otherwise stated, the population underlying each of the figures and tables covers adults aged 16-65.

This report has StatLinks for tables and graphs at the end of the chapters. To download the matching Excel® spreadsheet, just type the link into your Internet browser, or click on the link from the e-book version.

For consistency, national and subnational entities are referred to as “countries” and “economies”, respectively, in the whole publication. Territorial and subnational entities are referred to throughout the publication by their subnational name and country: England (United Kingdom) and the Flemish Region (Belgium).

Following guidelines established by the OECD Chief Statistician, in consultation with the Bureau of the Committee on Statistics and Statistical Policy, a visual separation between the data from subnational entities and the data from national entities is introduced, to the extent possible, in all figures and tables in this report. In Annex tables, results for subnational entities are reported separately, after the results for national entities. In figures, the names of subnational entities are reported in blue.

This report features data from 31 countries and economies: Austria, Canada, Chile, Croatia, Czechia, Denmark, England (United Kingdom), Estonia, Finland, the Flemish Region (Belgium), France, Germany, Hungary, Ireland, Israel, Italy, Japan, Korea, Latvia, Lithuania, the Netherlands, New Zealand, Norway, Poland, Portugal, Singapore, the Slovak Republic, Spain, Sweden, Switzerland, the United States.

The OECD average corresponds to the arithmetic mean of the respective estimates for all OECD countries and economies for which data are available. It was calculated for most indicators presented in this report. The following countries and economies are included in the OECD average: Austria, Canada, Chile, Czechia, Denmark, England (United Kingdom), Estonia, Finland, the Flemish Region (Belgium), France, Germany, Hungary, Ireland, Israel, Italy, Japan, Korea, Latvia, Lithuania, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden, Switzerland, the United States.

This report discusses only statistically significant differences or changes. These are denoted in darker colours in figures and bold font in tables. Unless otherwise specified, the significance level is set to 5%.

The details of the technical standards guiding the design and implementation of the 2023 Survey of Adult Skills can be found at https://www.oecd.org/en/about/programmes/piaac/piaac-data.html#manuals. Information regarding the design, methodology and implementation of the Survey of Adult Skills can be found in summary form in the Survey of Adult Skills 2023 Reader’s Companion (OECD, 2024[2])and, in detail, in the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]).

[2] OECD (2024), Survey of Adult Skills 2023: Reader’s Companion, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/3639d1e2-en.

[1] OECD (forthcoming), Survey of Adult Skills 2023 Technical Report, OECD Publishing, Paris.