Review by Adriana Albuquerque
The Programme for International Student Assessment (PISA) is a triannual survey to 15 year-old students from various countries in the world (in 2022, 81 countries and economies participated), whose goal is to evaluate the knowledge and skills acquired so far in school for a full participation in social and economic life. Students are evaluated in three domains: mathematics, science, and reading and they fill an additional characterization survey about their attitudes, beliefs, family lives and school experiences. PISA also collects information next to school principals about their organization’s management and environment.
Five volumes about the last edition of PISA (2022) – which counted with the participation of 690.000 students across the world – are scheduled to be launched. This text reviews the first two published reports – The State of Learning and Equity in Education (Vol. I) and Learning During – and From – Disruption (Vol. II). We will highlight the most relevant results for the analysis of inequalities in academic outcomes, from a comparative perspective between Portugal and the member / partner countries of the Organisation for Cooperation and Economic Development (OECD), as well as from a temporally dynamic perspective – in fact, this survey’s multiple editions make it a fundamental instrument for assessing the evolution of educational inequalities at a global level.
We begin with the first volume, which essentially contains data about student outcomes.[1] Concerning global average performance, Portugal presents scores within the OECD average in 2022 in Mathematics (472), Reading (477), and Science (484), being one of only five countries where this happens (alongside Lithuania, France, Spain, and Hungary). There were 25 countries where students obtained scores above the OECD average in at least one domain; the remaining 51 countries showed outcomes significantly below the average in at least two domains. Another important fact concerns the percentage of students with “top” and “bottom” performances relative to the OECD average (13,7% and 16,4%, respectively). In 2022, Portugal has less excellent students in at least one domain (-3,6 p.p.). At the same time, it is amongst the 25 countries with the least students at risk of general failure (-2,6 p.p.), which points to a non-polarised performance distribution.
From a diachronic perspective, the global drop in performance is clear in all three domains, especially on the short-term: in OECD countries, there is a negative average variation in Mathematics, Reading and Science of -15, -10 and -2 scores, respectively, since 2018; on the long-term, the decline is less accentuated in Mathematics and Reading, but worse in Science (7, -4 and -7, respectively). In this scenario, Portugal is an interesting case: it presents a more intense drop than the OECD average in short-term outcomes (of -21, -15 e -7 scores, respectively), whilst demonstrating a liquid positive evolution since the year of its first PISA participation (2000) of +8, +7 and +5, respectively.
How did this general decline in results affect students of different socioeconomic status? Compared to 2018, the report shows that disadvantaged students in the OECD registered an average drop of -17 scores in Mathematics, whilst their advantaged peers suffered a drop of -10 scores. In Portugal, disadvantaged students’ Mathematics’ performance was penalized in the same proportion of the global average, whilst advantaged students dropped twice as much as their OECD peers (-20 scores). Hence, although advantaged students in Portugal have a bigger lead in Mathematics results, compared to the OECD average (+101 scores, compared to +91, respectively), and despite the bigger predictive weight of social origins on student outcomes (18,2% of variance is explained by student socioeconomic status, compared to a 15,5% average), the short-term drop in Mathematics performance did not rely on the worsening of disadvantaged students’ outcomes.
Repeating the analysis for gender inequalities, we see that, on average, the gap grew from 2018 to 2022, in Mathematics (+4 scores) and Reading (+5 scores), strengthening the advantage of boys (477 scores, compared to 468 for girls) and girls (488 scores, compared to 464 of boys), respectively; only Science shows a levelling of averages by gender (485 scores). In Portugal, gender inequality also grew in Mathematics and Reading (+2 and + 3, respectively), even if to a lesser degree, following the trend verified in OECD: a masculine advantage in Mathematics (477 scores, compared to 467 for girls) and a feminine advantage in Reading (487 scores, compared to 466 of boys). Like in OECD, Portugal has also witnessed the near elimination of the gender gap in Science.
Concerning migratory inequalities, we see that in most countries, students with an immigrant origin experience a softer decline in Mathematics’ performance than their peers without an immigrant background, between 2018 and 2022. In Portugal, whilst the former decreased -7 scores (a non-significant variation), the latter decreased -20 scores, i.e., almost triple their peers. Thus, students in Portugal perform within the OECD average in Mathematics, in 2022, independently of their migratory / generational status being none (477 scores), “second generation” (461 scores) or “first generation” (434 scores). However, it is important to note that there are still severe inequalities between students with and without a migratory background even after the introduction of statistical controls for socioeconomic status and language spoken at home. Whereas in OECD the average difference in Mathematics scores is -5 scores, sanctioning immigrant students, in Portugal their disadvantage is four times higher than in other countries (-20 scores).
The second volume switches analytic lenses, from student results to system-level and school-level outcomes – namely, their differential ability for resilience between 2018 and 2022, i.e., to sustain within or above-average results in (i) Mathematics performance; (ii) socioeconomic equity of outcomes; and (iii) student well-being in school. Only 25 educational systems showed resilience in at least one of these domains and only 4 showed it in all domains: Japan, South Korea, Lithuania, and Chinese Taipei. Portugal is among the biggest set of resilient countries (15) that was so only for student well-being – the index of school belonging was 0,08 points compared to a -0,01 average in OECD. When searching for explanations for this inequality in education systems’ resilience, it is impossible not to look at the impact of the COVID-19 pandemic and, above all, the resulting confinements. In the most resilient countries, it stands out that a bigger proportion of students experienced shorter confinements (3 months or less) compared to the OECD average (53,6%), as well as lower than average indexes of problems with remote learning (-0,01). In Portugal, more students than the average reported that their school was closed for a short period of time (58,1%), as well as considerably less problems with remote learning (-0,19).
Another key to understanding the greater resilience to the pandemic of some educational systems seems to be differences in their degrees of vertical stratification – including the separation between distinct educational levels, but also the practice of grade repetition – and horizontal stratification – involving the early tracking of students into different instructional programs and sorting students into homogenous groups (the so called ‘ability grouping’) within the same grades. In fact, Austria, Germany, and Hong Kong are the only systems considered “resilient” that apply more than one of the main educational stratification measures (ability grouping, instructional tracking, and grade repetition). The most prevalent set of systems can be considered comprehensive, i.e., where students are in heterogenous classrooms with the same curriculum and usually continue to the next grade – it is also in this group that we find most “resilient” countries, namely, three out of the four that are so in all domains (Japan, South Korea, and Chinese Taipei).
Mechanisms of vertical and horizontal stratification show signs of decline in most educational systems since 2018, and Portugal is no exception, especially with respect to grade repetition: in 2022, 17,2% of 15-year-old students in the country had repeated a grade at least once, compared to 9,4% in OECD (-9,4 p.p. and -1,9 p.p. compared to 2018, respectively). Part of the explanation comes from the exceptional nature of the pandemic period, but not exclusively. Considering the high permeability of international education systems to PISA results (Bonal & Tarabini, 2013), it is not unreasonable to think that the past editions of the survey created pressure in the international political agenda to reduce system-level stratification in education, given the convincing demonstration of its negative impacts on student outcomes and equity. The 2022 edition strengthens this evidence: average Mathematics outcomes are negatively correlated with the proportion of students put in ability groups within their classroom in all lessons, the between-schools index of isolation of socially advantaged students, and the proportion of students that has repeated a grade at least once. The importance of valuing diversity within the classroom is emphasized in Portugal, given that peer-to-peer tutoring – which can only happen between students with different skill levels – is the only study support measure with a significant positive effect on Mathematics performance after controlling for socioeconomic origin (+13 scores, compared to +5 scores in OCDE). Good news, since Portugal is among the countries where this practice increased the most since 2018 (+30,6 p.p., compared to +3,1 p.p. in OCDE).
Although performance increases in systems with instructional tracking, its negative effect on the socioeconomic equity index shows that in these countries, the global improvement in outcomes is done at the expense of the disproportionate streaming of disadvantaged students into vocational tracks, as has been amply diagnosed in literature (Boone & van Houtte, 2013). Predictably, all academic and social between-school segregation measures, as well as the proportion of students that has repeated a grade at least once, have a negative impact on equity. On the contrary, age at first selection into a different educational programme is positively associated with equity indexes.
On equity, part of the explanation for academic and social segregation between schools lies in competition dynamics, namely, between public and private sector. PISA 2022 shows these subsystems have different recruitment pools in most OECD countries, including Portugal, with private schools absorbing considerably more high socioeconomic status students with no migratory background. Private schools also report significantly less lack of material and human resources than public schools. However, after controlling for student social origins, public schools on average (and in 22 participant countries) obtain higher Mathematics scores than private schools (+11 scores). Only in 17 countries do private schools produce added value in student performance after controlling their social composition. Portugal is among the most prevalent set of countries (29) where the public-private gap in performance is not significant. In fact, the added value of the private sector happens only in schools with the most socially privileged compositions, obtaining on average +7 Mathematics scores. This dwarfs the added value produced by public schools with socially disadvantaged compositions, where students have +12 scores than they would in a socially similar private school.
In many educational systems, the pandemic was one of the causes for the acceleration of (i) digital transition and (ii) student-centred learning methodologies. Concerning the digital transition in education, we see that most 15-year-olds in OECD feels confident using video communication programmes (77%) and motivating themselves to do schoolwork (58,1%); Portugal is above the average in both indicators (84,6% and 65,6%, respectively). In fact, Portugal has made a leap forward in the average number of computers available per student in schools, between 2018 and 2022, going from a 0,39 ratio to 0,69, shortening the distance from OECD (whose ratio is currently around 0,81). Despite this, digital resources available in schools are considered inadequate or low quality by 39,5% of Portuguese principals, compared to a 24,9% OECD average. Portuguese schools also have a lower-than-average index of preparedness for digital learning (-0,18 compared to -0,02).[2]
The integration of digital technologies for learning raises new challenges, namely, in managing a classroom’s climate and its consequences on student focus. In OECD, only around 34,6% of students report never feeling distracted in Mathematics class due to using digital devices, whereas in Portugal this figure is even lower (26,9%). This is one of the disciplinary climate factors with the strongest association with Mathematics outcomes (decreasing -8 scores) and in Portugal (decreasing -13 scores). The solution seems to be in the moderate use of digital tools in school. In fact, prohibiting these devices may answer the “digital distraction” problem, but it does so at the cost of academic outcomes. Counterintuitively, using digital devices for leisure in school is better for Mathematics results than not using them, within certain limits: up to 2 hours per day in OECD and up to 3 hours in Portugal (with rises of +11 scores in both cases). Using these devices for learning brings benefits up to 7 daily hours in OECD (+9 scores) and up to 3 daily hours in Portugal (+5 scores), reaching the added value ceiling at the 1 daily hour in OECD (481 scores, i.e., + 25 compared to no hours) and Portugal (485 scores, i.e., + 15 compared to no hours).
We conclude with the impact of teacher expectations on student outcomes – specifically, the way in which these are perceived by students. A significant relationship was identified between perceived teacher support and: (i) increased student confidence in own ability to learn autonomously; (ii) decreased Mathematics anxiety; and (iii) increased Mathematics performance. Portugal was the 7th country where the most students reported feeling supported by their teachers (75,1%, compared to 67% on average). However, this perception has worsened between 2018 and 2022 in the OECD and Portugal. Still, students in Portugal continue to evaluate teacher support more positively than the average in all domains: (i) teacher shows an interest in every student’s learning (74,7%, +11,5 p.p.); (ii) teacher gives extra help when students need it (78,9%, +8,7 p.p.); (iii) teacher helps students with their learning (80,8%, +9,2 p.p.); (iv) teacher continues teaching until students understand (74,6%, + 11,3 p.p.).
Despite this, it is important to highlight inequalities in perceived teacher support by different students. On average, boys, students of disadvantaged socioeconomic status and with an immigrant background feel more supported (according to the teacher support index, +0,05, +0,07 and +0,04, respectively), than girls, socially advantaged and no immigrant background students. In Portugal, only migratory status has a significant effect, and it goes against the average OECD trend: immigrant students feel significantly less supported by teachers than their “native” peers (-0,12), the only OECD country besides Estonia where this happens (-0,17).
References
Bonal, X., & Tarabini, A. (2013). The Role of PISA in Shaping Hegemonic Educational Discourses, Policies and Practices: The Case of Spain. Research in Comparative and International Education, 8(3), 335-341. https://doi.org/10.2304/rcie.2013.8.3.335.
Boone, S., & van Houtte, M. (2013). In search of the mechanisms conducive to class differentials in educational choice: a mixed method research. The Sociological Review, 61(3), 549–572. https://onlinelibrary.wiley.com/doi/10.1111/1467-954X.12031
End notes
[1] PISA scores do not have a substantive meaning, i.e., they do not represent a scale with minimum and maximum values. Instead, they result from a standardization process based on the normal distribution, which results in averages of around 500 points and standard deviations of about 100 points. This allows for direct between-country and between-edition data comparisons.
[2] This index measures teachers’ digital skills, time available to integrate digital technologies into their instructional practices, available resources to learn how to use digital devices, school incentives and support to do so and the existence of sufficient technical assistance staff in schools.
Published on 24/04/2024
