There’s an old joke. A very sick man is being operated on, but the surgeon is unable to save him and the man dies. The surgeon then has the unpleasant task of informing the waiting wife.
“Tell me Doctor, how is my husband?”
“Well the good news is, his condition is stable.”
The joke came to mind this morning, with the reporting of the just-released 2024 NAPLAN results. The reports everywhere are of the grimness, of “one in three students not meeting standards“, of the “NAPLAN mediocrity“, of “the new lows“, of the “epic fail“, and on and on and on. ACARA, however, had a different take.
ACARA issued a news item and a media release. The media release was titled,
Latest data shows 2024 NAPLAN national results broadly stable
ACARA’s media release went on at length, embellished with quotes from ACARA CEO, Stephen “no errors” Gniel:
… overall the results were broadly stable.
… remaining relatively stable in in [sic] Year 7 … and then dropping …
… the results were broadly stable …
The changes … are also broadly stable …
… the size of the changes is entirely consistent with the results remaining stable …
“The fact that the results of Australian students have remained stable through these changes, with no significant difference in average scores, is an important achievement.”
Yep, stable. And dead.
For anyone with any sense of the debacle that is Australian education, none of this is remotely a surprise. Readers can work through the details on ACARA’s website. The Guardian report has some pretty graphs and a number of reports have focussed on the elephant facts, that if the kid has “foreign language” parents or if the kid doesn’t go to Shitkicker Primary then he’s got a significantly better chance of learning something. The best response is courtesy of the Grattan guys. In brief: don’t teach shit. (This is a minor paraphrasing.)
We won’t bother with the new details here. We’ll simply make a few points, which we’ve made about a million times previously but that are routinely ignored.
No one knows what NAPLAN is testing
The tests are secret. No except us seems to much care that the tests are secret, but they are secret. Which means that no one can determine whether or not the 2024 NAPLAN tests were crap.
The 2024 NAPLAN tests were crap
How can we be sure? It is true, without seeing the tests we cannot be 100% sure. Let’s put it at 99%.
The NAPLAN numeracy tests have always been crap. They test word-swamped trivia, are poorly written and almost invariably contain the cherries of bad ambiguity and outright error.
Whatever the reported NAPLAN results, the actual results are worse
The NAPLAN numeracy tests are not simply bad tests, they are badly weak tests, semi-conforming to a badly weak curriculum. Any parent who believes that their kid in the “strong” band is strong is utterly deluded. As for the third of kids below “strong”, who are “developing” or who “need additional support”, they are well on the way to being completely screwed. This is ACARA’s definition:
Developing: the student’s result indicates that they are working towards expectations at the time of testing.
Orwell would have been proud of this one. The tests do not provide any evidence that the student is working towards expectations, that he is “developing” in any proper sense. The odds are, he is simply floundering.
There’s nothing else to say. ACARA’s media release has, however, inspired us to come up with a new acronym, which we predict will see plenty of use: SSAFU.
I just happened to see Barney Glover (Jobs and Skills Australia) at the National Press Cub address. Barney was a student of Bruce Craven at Melbourne in the 90’s (PhD in ’93) and clearly has been a very successful political operator in academia (something I never achieved)
The NAPLAN discussion may not appear relevant to that piece of information but what he said definitely was of relevance to Australian education, of which NAPLAN is a minor indicator. To paraphrase, he indicated there is a large (and growing) number of students with education and skills deficits below that required for participation in the Australian economy. And that there was no indication of this changing in immediate future.
Ever the optimiser, Barney indicated that a large skilled migration program was the most efficient input to grow the economy. Basically indicating that at least one generation of students will be surplus to an optimal solution to growing the economy.
The NAPLAN scores probably fit with his assessment of the incoming workforce
I know you said a third of students were below the standards, but the real triumph was
“Oceania youth league reports two thirds achieving standards!”
Somehow I think “import more Chinese and Indians” is not the optimal strategy.
George Megalogenis notes the risk to social cohesion of bringing in many migrants who are richer and better educated than the native population – the modern strategy.
Yes, it is a very dubious solution to the wrong problem.
I taught a year 8 class at a well respected Gummint school in 2019. I looked at their Naplan results from year 7. Most were listed as having acceptable achievement. NONE of them knew any maths at all. No doubt the fudging continues.
NAPLAN does not test mathematics. Or anything that matters.
But does NAPLAN correlate with something that is valid? And if yes, does it matter how bad it is? All this comparison data doesn’t seem to have made much difference in policy terms.
Yes, it must correlate. But:
a) The correlation will be weak, since numeracy ≠ mathematics.
b) The nature of the correlation is secret, since the tests are secret.
c) NAPLAN and the AC are the creations of the education academics and the educrats, who are the same clowns then called upon to fix things.
I have to get this off my chest, but why do Naplan insist on reporting a confidence interval for their results?
https://www.acara.edu.au/reporting/national-report-on-schooling-in-australia/naplan-national-results
Click on “achievement by state” and you will see nice tabulation, with average NAPLAN score for each state and Australia as a whole, lovingly followed by a confidence interval in brackets. But what on earth does this mean? I could see a reason to report a standard deviation or an interquartile range – but a confidence interval?
Maybe it’s subliminal messaging? They’re trying to suggest we should have “confidence” in the results?
But seriously, it’s a pretty good question. I have no idea what it means.
Confidence intervals are calculated from sample averages, sample sizes and sample standard deviations. They provide one way to compare results from different samples.
So what are the “samples” here?
Not all students take the test. So the samples are the the samples of students in a particular category who took the test. For example, you might want to compare the average results of students from group A with the results of students in group B.
What? You’re talking about NAPLAN? What “samples” are you talking about?
The students who take the test are a sample of the students who are eligible to take the test. For example, not all Year 9 students take Year 9 NAPLAN tests. Of course this is not a random sample, but it is a sample of the population of year 9 students.
Jesus.
What fraction of 2024 Year 9 students do you guess took the 2024 Year 9 NAPLAN test? 80%? 90%?
Are you really arguing this is an appropriate place to be talking confidence intervals? Do you really think that’s what ACARA has done?
I was just trying to answer the question “So what are the “samples” here?”
Participation for Year 9 Numeracy was 88.6% across Australia. I expect that there would be substantial variation between schools across Australia.
If your sample is 90% of the population, does it make any proper sense to calculate a confidence interval? Do you think that is what ACARA has done in this instance?
CI’s give you a way of identifying significant differences between two groups of students. If the CIs do not overlap, then there is a significant difference between the average scores in the two groups. On the other hand, if there is some overlap, no such conclusion can be drawn; it depends on the amount of overlap (- a point which is often not appreciated).
Christ, Terry. This was started by notateacher asking why ACARA reported the results with confidence intervals. Do you have a plausible answer for this or not?
Marty, CIs provide a quick way to see a significant difference between two samples – if they are disjoint. I am sorry that I cannot think of a better answer to the question.
I’m not convinced that this is what ACARA is doing or, if they are doing it, why they are doing it.
Yes well, I certainly wasn’t expecting this discussion.
I am not going to attempt to define a confidence interval but simply say the calculation assumes that the sample <<< the population. It is just calculated using the sample mean and standard deviation, the z score and the sample size – it is indifferent to the population size. That is the calculated sample mean doesn't constrain the true population mean beyond the confines of statistical theory.
If the sample mean included the entire population you could still plug the data into the confidence interval calculator and it will spit out a number, but the number would be meaningless. In this case the sample mean = the population mean.
If your sample was 99% of the population, then your sample mean wouldn't precisely equal your population mean, but the difference would be insignificant because the sample mean would tightly constrain what the population mean could be.
If you have a situation when your sample was 88% of the population and you assume no biases in the missing data, then you could you use a normal distribution to estimate a confidence interval of the mean of the missing data and use that to calculate the range the true population mean should fall between.
And the reason you wouldn’t do that is no one would assume the people who aren’t taking NAPLAN are an unbiased sample. The assumption would surely be they are largely people disengaged from school or who are worried about the test, plus a few that are genuinely sick.
Thanks, nat. Exactly my thoughts. In fact I have reasons to suspect something else is going on.
Right. The 88% “sample” is most definitely not random.
In most reports that involve sampling, the samples are not random.
BTW, the definition of a random sample in many text books is wrong.
There are correction factors that can be used when the sample is a large part of a finite population. I don’t know if these are built into the calculations in NAPLAN.
Terry, has it occurred to you that ACARA may be just screwing up?
No – I have not looked at NAPLAN reports.
I retired from an academic-ish position last year. That is to say, the institution at which I worked was dignified by the name of “University”, but that was its best feature. Everybody knows that students leave school with pitiful to no mathematical skills, and so it’s up to the universities – already suffering from overcrowded syllabi, job uncertainty and wage theft – to bring the students up to a standard where they might be employable. And this is basically impossible. High school teachers complain that primary schools don’t teach the basics, and primary school teachers have such a stupidly impossible job managing classes with ever dwindling resources they they are lucky to survive in one piece, let alone try to teach. I have the highest respect for primary teachers, but I have also taught secondary pre-service teachers, and in the end I basically gave up. I couldn’t teach mathematics, so I taught mathematical recipes instead.
So what’s all this to do with NAPLAN? Simply that if the folk at ACARA could take off their rose-tinted spectacles, and move from whatever alternative reality they inhabit to the real world, they might see that there is a lot wrong with every aspect of our educational systems. Then they could aim at, you know, beginning to find solutions, rather than patting themselves on their simian backs for jobs apparently well done.
I think I’m stating the obvious here, but I felt a rant coming on, and well, better out than in!
Also, Barney’s idea of importing good students may not be optimal, but if we’re to have a properly trained and expert workforce, it may not be such a bad stopgap solution. (Stopgap = about 30 years).
Not a bad rant. Now that you have time on your hands, wanna take over the blog?
Yes well to Barney it IS the optimal solution to just bring in immigrants to fill skill gaps. He literally has his integer programming guys allocating resources (he didn’t announce that bit, but that’s his thing and it was a non-technical talk).
He was pretty blithe about how the school systems suck and won’t improve soon- the journos didn’t even blink. Implied in his statement was that if there happens to be a couple of million students over 20 years consigned to low-pay, marginal work ; being shelf-stackers, drive-thru attendants, age care, baristas, truckers etc. then that’s just fine as they’re workers deserving of respect like any other (which is true). I just found it depressing and obscene that in less than 2 minutes so many people were marginalised rather than declaring a state of emergency in the education system and doing something….anything.
Enjoy your retirement Alisdair. I’m still 7 years off. At my “academic” institution (for “academic”- read soulless, self-promoting, managerial monolith that degrades young people and staff equally) it’s worth noting that several academics from overseas have sent their offspring to their home countries rather than pretend that the swill we offer at secondary level will allow a decent education ( or getting a second mortgage to send them to a private school)
Enough rant, more sleep
Barney’s “solution” is to a different problem. It’s a mad take.
Just a comment from far away: Australia wins 53 medals in total, 18 of them golden, bravo. That’s what the public is interested in, so that’s what really counts.
But there seems to be a correlation between olympic medals and basic school skills.
There does?
Your link to the Grattan article is paywalled, but it is also on their website sans paywall:
Ah, thanks very much, MW. I’ve also replaced the two Grattan links in the post.