NAPLAN’s Numerological Numeracy

This year Australia celebrates ten years of NAPLAN testing, and Australians can ponder the results. Numerous media outlets have reported “a 2.55% increase in numeracy” over the ten years. This is accompanied by a 400% increase in the unintended irony of Australian education journalism.

What is the origin of that 2.55% and precisely what does it mean to have “an increase in numeracy” by that amount? Yes, yes, it clearly means “bugger all”, but bugger all of what? It is a safe bet that no one reporting the percentage has a clue, and it is not easy to determine.

The media appear to have taken the percentage from a media release from Simon Birmingham, the Federal Education and Training Minister. (Birmingham, it should be noted, is one of the better ministers in the loathsome Liberal government; he is merely hopeless rather than malevolent.) Attempting to decipher that 2.55%, it seems to refer to the “% average change in NAPLAN mean scale score [from 2008 to 2017], average for domains across year levels”. Whatever that means.

ACARA, the administrators of NAPLAN, issued their own media release on the 2017 NAPLAN results. This release does not quote any percentages but indicates that the “2107 summary information” can be found at the the NAPLAN reports page. Two weeks after ACARA’s media release, no such information is contained on or linked on that page, nor on the page titled NAPLAN 2017 summary results. Both pages link to a glossary, to explain “mean scale score”, which in turn explains nothing. The 2016 NAPLAN National Report contains the expression 207 times, without once even pretending to explain what it means. The 609-page Technical Report from 2015 (the latest available on ACARA’s website) appears to contain the explanation, though the precise expression is never used and nothing remotely resembling a user-friendly summary is included.

To put it very briefly, each student’s submitted test is given a “scaled score”. One purpose of this is to be able to compare tests and test scores from different years. The statistical process is massively complicated and in particular it includes a weighting for the “difficulty” of each test question. There is plenty that could be queried here, particularly given ACARA’s peculiar habit of including test questions that are so difficult they can’t be answered. But, for now, we’ll accept those scaled scores as a thing. Then, for example, the national average for 2008 Year 3 numeracy scaled scores was 396.9. This increased to 402.0 in 2016, amounting to a percentage increase of 1.29%. The average percentage increases from 2008 to 2017 can then be further averaged over the four year levels, and (we think) this results in that magical 2.55%.

It is anybody’s guess whether that “2.55% increase in numeracy” corresponds to anything real, but the reporting of the figure is simply hilarious. Numeracy, to the very little extent it means anything, refers to the ability to apply mathematics effectively in the real world. To then report on numeracy in such a manner, with a who-the hell-cares free-floating percentage is beyond ironic; it’s perfect.

But of course the stenographic reportage is just a side issue. The main point is that there is no evidence that ten years of NAPLAN testing, and ten years of shoving numeracy down teachers’ and students’ throats, has made one iota of difference.

Obtuse Triangles

Whatever the merits of undertaking a line by line critique of the Australian Curriculum, it would take a long time, it would be boring and it would probably overshadow the large, systemic problems. (Also, no one in power would take any notice, though that has never really slowed us down.) Still, the details should not be ignored, and we’ll consider here one of the gems of Homer Simpson cluelessness.

In 2010, Burkard Polster and I wrote an Age newspaper column about a draft of the Australian Curriculum. We focused on one line of the draft, an “elaboration” of Pythagoras’s Theorem:

recognising that right-angled triangle calculations may generate results that can be integral, fractional or irrational numbers known as surds

Though much can be said about this line, the most important thing to say is that it is wrong. Seven years later, the line is still in the Australian Curriculum, essentially unaltered, and it is still wrong.

OK, perhaps the line isn’t wrong. Depending upon one’s reading, it could instead be meaningless. Or trivial. But that’s it: wrong and meaningless and trivial are the only options.

The weird grammar and punctuation is standard for the Australian Curriculum. It takes a special lack of effort, however, to produce phrases such as “right-angled triangle calculations” and “generate results”. Any student who offered up such vague nonsense in an essay would know to expect big red strokes and a lousy grade. Still, we can take a guess at the intended meaning.

Pythagoras’s Theorem can naturally be introduced with 3-4-5 triangles and the like, with integer sidelengths. How does one then obtain irrational numbers? Well, “triangle calculations” on the triangle below can definitely “generate” irrational “results”:

Yeah, yeah, \pi is not a “surd”.  But of course we can replace each \pi by √7 or 1/7 or whatever, and get sidelengths of any type we want. These are hardly “triangle calculations”, however, and it makes the elaboration utterly trivial: fractions “generate” fractions, and irrationals “generate” irrationals. Well, um, wow.

We assume that the point of the elaboration is that if two sides of a right-angled triangle are integral then the third side “generated” need not be. So, the Curriculum writers presumably had in mind 1-1-√2 triangles and the like, where integers unavoidably lead us into the world of irrationals. Fair enough. But how, then, can we similarly obtain the promised (non-integral) fractional sidelengths? The answer is that we cannot.

It is of course notable that two sides of a right-angled triangle can be integral with the third side irrational. It is also notable, however, that two integral sides cannot result in the third side being a non-integral fraction. This is not difficult to prove, and makes a nice little exercise; the reader is invited to give a proof in the comments. The reader may also wish to forward their proof to ACARA, the producers of the Australian Curriculum.

How does such nonsense make it into a national curriculum? How does it then remain there, effectively unaltered, for seven years? True, our 2010 column wasn’t on the front of the New York Times. But still, in seven years did no one at ACARA ever get word of our criticism? Did no one else ever question the elaboration to anyone at ACARA?

But perhaps ACARA did become aware of our or others’ criticism, reread the elaboration, and decided “Yep, it’s just what we want”. It’s a depressing thought, but this seems as likely an explanation as any.

Accentuate the Negative

Each year about a million Australian school students are required to sit the Government’s NAPLAN tests. Produced by ACARA, the same outfit responsible for the stunning Australian Curriculum, these tests are expensive, annoying and pointless. In particular it is ridiculous for students to sit a numeracy test, rather than a test on arithmetic or more broadly on mathematics. It guarantees that the general focus will be wrong and that specific weirdnesses will abound. The 2017 NAPLAN tests, conducted last week, have not disappointed. Today, however, we have other concerns.

Wading into NAPLAN’s numeracy quagmire, one can often find a nugget or two of glowing wrongness. Here is a question from the 2017 Year 9 test:

In this inequality is a whole number.

\color{blue} \dfrac7{n} \boldsymbol{<} \dfrac57

What is the smallest possible value for n to make this inequality true?

The wording is appalling, classic NAPLAN. They could have simply asked:

What is the smallest whole number n for which \color{red} \dfrac7{n} \boldsymbol{<} \dfrac57\, ?

But of course the convoluted wording is the least of our concerns. The fundamental problem is that the use of the expression “whole number” is disastrous.

Mathematicians would avoid the expression “whole number”, but if pressed would most likely consider it a synonym for “integer”, as is done in the Australian Curriculum (scroll down) and some dictionaries. With this interpretation, where the negative integers are included, the above NAPLAN question obviously has no solution. Sometimes, including in, um, the Australian Curriculum (scroll down), “whole number” is used to refer to only the nonnegative integers or, rarely, to only the positive integers. With either of these interpretations the NAPLAN question is pretty nice, with a solution n = 10. But it remains the case that, at best, the expression “whole number” is irretrievably ambiguous and the NAPLAN question is fatally flawed.

Pointing out an error in a NAPLAN test is like pointing out one of Donald Trump’s lies: you feel you must, but doing so inevitably distracts from the overall climate of nonsense and nastiness. Still, one can hope that ACARA will be called on this, will publicly admit that they stuffed up, and will consider employing a competent mathematician to vet future questions. Unfortunately, ACARA is just about as inviting of criticism and as open to admitting error as Donald Trump.