TIMSS, Australia and the Iraqi Army

UPDATE (21/01/25)

No big deal, but I just stumbled across an article in the online magazine Teacher, a joint article by Wernert and Teacher editor, Jo Earp. Written to accompany ACER’s release of Australia’s TIMSS results, the article is similar to Wernert’s Conversation piece but is longer and sillier. Why these people see it appropriate to be cheerleading rather than reporting, God only knows.

******************************************************************

The 2023 Trends in International Mathematics and Science Study results came out last week, and it appears to have Rorschached people. The Australian cried doom and gloom, including CIS’s Glenn Fahey doing the right wing thing of suggesting that we might want to teach young kids some maths. The Age went with Victoria lagging behind other states before repackaging the report to a more Australian line. The Herald Sun had some STEM guy declare, to the surprise of no one, that what we need is more STEM subjects. The SMH and the ABC went with the persistent boy-girl gap, ABC radio following up with a mathematics professor declaring that what we need is more Eugenia Cheng. Yeah, maybe, although Eugenia would not be my top recommendation. Perhaps, rather, it is time to conclude that the years of such crusading hasn’t helped anyone, including the girls.

Pretty much on their lonesome, the Guardian was cheering how good were the Australian TIMSS results:

Australian primary students score best ever result in global maths and science test, despite pandemic disruption

Timss report finds year 4 students ranked equal fifth internationally out of 58 countries, up from ninth in 2019

Is that true? Well, sort of:

The report found Australian year 4 students, whose first years of schooling occurred doing Covid, ranked equal fifth internationally out of 58 countries, and equal 15th in mathematics.

OK, so the Year 4 mathematics result of 15th (ish) is not as impressive as the overall result. Still, even Australia’s Year 4 maths “score” went up, from 516 in 2019 to 525 in 2023. So why was only the Guardian so chirpy? Why were they the outlier? Perhaps because enough of the other reporters were, for once, not content to stenograph.

TIMSS in Australia is handled by ACER and, whatever their other failings, ACER seems to be good at this test management stuff. Accompanying the Australian TIMSS Report and the highlights, however, ACER issued a media release:

Year 4 students hit new high in global assessment, despite pandemic

Sound familiar? The media release quoted lead report author, Nicole Wernert, who also had a Conversation article:

Year 4 students just achieved their best ever result in a major maths and science test

Yes, there is a theme. Wernert notes the boy-girl gap and other concerns but ACER’s media release and Wernert’s Conversation article are decidedly, and inappropriately, upbeat. The Guardian then, simply, seems to have bought it.

The whole thing is very ACARA-ish, featuring determined cheerleading where ACER’s only proper role is dispassionate reporting. Then, given there is much less to cheer about than ACER is pretending, nonsense inevitably creeps in:

Year 8 students have maintained the achievement levels set in previous years, with similar results to those recorded in 2019.

Australia’s Year 8 mathematics declined from 517 in 2019 to 509 in 2023, almost exactly mirroring the Year 4 increase that ACER is so happy to shout from the rooftops (p 53). So, not that “similar” and not exactly “maintained”.

The strength of the improvement at Year 4 in science suggests efforts to promote STEM in early education are beginning to see results.

It is hardly a controlled experiment, and there are hidden uh-ohs to consider, but maybe.

We might see this flow through to the middle years when TIMSS is next tested in 2027. This is when the Year 4 group from 2023 will be in Year 8.

Yes, and we might not. Anyone here a betting man?

By 2027, we would also expect to see the effect of the updated Australian Curriculum that began to roll out in 2023. This has a greater focus on mastering key skills in maths.

Wernert may know her stats but she clearly doesn’t know a thing about the godawful Australian Curriculum. But that’s the thing when one starts barracking instead of sticking to scientisting: one is tempted to say very stupid things.

********************************************************************

Still, on the face of it, the Year 4 mathematics results have improved and, whatever else is going on with TIMSS, this would be a good thing. But there is other stuff going on, even beyond the Year 8 decline, and there are three points worth making, all of them ignored or undersold in last week’s coverage.

First of all, and as always, no one gives a stuff what the test is testing. Is TIMSS worth taking seriously? Does TIMSS test mathematics or is it, in the tradition of PISA and NAPLAN, real-world numeracy twaddle? Pretty much no one knows, and pretty much no one cares; they only know that 525 is greater than 516 and that’ll do ’em. Education reporters are not a very inquisitive lot.

As it happens, TIMSS is a good mathematics test, as evidence by the released questions from 2019. But no one cares. The substantially meaningless outcomes of PISA and NAPLAN will be reported and will be consumed in exactly the same manner.

Secondly, no one knows what the “scores” 525 and 516 and so forth mean; and, again, pretty much no one cares. But, as Franz Lemmermeyer noted when we were doing this four years ago, the numbers mean something. It is very difficult to determine precisely what, however.

The ballpark idea is that, for each TIMSS test, students’ scores are scaled to have a mean of 500 and a standard deviation of 100. This was how things were set up in 1995, when TIMSS began. But of course students on average may be improving (or may be getting worse) with each test season, and subsequent scores should reflect such changes. To this end, the scales are adjusted with each report. Precisely how this adjustment is determined must be outlined somewhere in the jungle of the 2019 Technical Report, but we have not attempted to find it; the underlying idea is outlined in England’s 2023 report (p 35). In any case, the adjustments from test to test tend to be minor and it is reasonable to claim/presume that a score increase from 516 to 525 indicates some improvement, both relative to the new mean and absolutely.

Still, this does not tell us what these scores mean, the indication they provide of what kids can or cannot do. For that, one has to look at the test questions and at countries’ performances on these questions. The 2023 England report contains a few released items (pp 153-157), but with no indication of performance. Instead, we can look to the 2019 released items. Below are two (harder) items from the 2019 Year 4 and Year 8 tests, respectively.

The Year 4 question was answered correctly by 53% of the tested Australian students, and the Year 8 question by 21%. One can decide whether such success rates on such questions are acceptable.

********************************************************************

Acceptable or not, there is the question of Australia’s comparative success. Recall that The Guardian was echoing ACER’s trumpeting of Australia’s climb up the Year 4 league tables. This comparative success is the third and final, and most important point.

As always, we can compare to Singapore, which, unlike ACARA, we can do honestly. On the two questions above, for example, Singapore students’ success rates were, in turn, 86% and 46%, dramatically higher than Australia’s. This is consistent throughout the released 2019 questions, and with the other Asian “tigers” not far behind Singapore. This is the real story of TIMSS.

The Australian and Fahey and others made the point, but not sufficiently strongly or sufficiently clearly. It is not simply that the usual Asian suspects performed better than Australia; it is that they blew Australia away. In 2023, Singapore’s Year 4 and year 8 mathematics scores were, respectively, 615 and 605, about a full standard deviation above Australia. The other Asians were not far behind: they are the magenta dots in the plots above, with Australia the red dots.

This standard deviation gap kicks in most, and very dramatically, at the top end. In her Conversation piece, Wernert was patting Australia’s high achievers on the back:

A lot of the improvement at Year 4 was due to a significant increase in the percentage of very high performing students. These are students who perform at advanced levels [above 625] for their year level.

In 2023, one in every eight Australian Year 4 students were very high performers in maths …

Yep, well done you one in every eight Australian kids. In Singapore, it was one in every two kids.

********************************************************************

In 1990, Iraq invaded Kuwait, after which Bush and his buddies prepared to respond. At this time, as well as questions about the morality of the planned military response, there was plenty of discussion about the potential difficulty of the response, about the strength of the Iraqi army. In this context, it was common to note that Iraq had the “fourth largest army” in the world. This was true, although what it meant was not typically spelled out. And, more to the point, “fourth” was leagues below the size and the power of the top three. The war was quick. In the important reality at hand, the Iraqi army didn’t rate.

Australian mathematics education is, at its best, on a par with the Iraqi army. That’s what it aspires to, what it applauds when it succeeds. In the important reality at hand, it doesn’t rate. It doesn’t even try.