Tony Guttmann, AKA Mr. Very Big, is a legend of Australian mathematics and of Australian mathematics education. He is a fellow of the AAS and pretty much everything else. Tony was the hero of Curriculum War 1 and was on the front lines of Curriculum War 2. Tony signed the recent mathematicians’ open letter on the errors in the 2022 VCE mathematics exams. Last week, Tony also submitted a letter to The Age, which The Age has so far declined to publish. Tony has given us permission to post his letter here. Tony’s letter follows.
Dear Editor,
Once again, the VCE mathematics examination has been plagued with errors. In my experience this has been a recurring issue for at least the past 40 years, and is as preventable as it is inexcusable.
Last year the Victorian Curriculum and Assessment Authority commissioned a report from one of the big 4 accounting firms and other unnamed educational authorities and found themselves exonerated!
The solution is to get a professional mathematician (or ideally two) to check the examination papers before they are issued. How hard is that? Yet for nearly half a century the VCAA and its predecessors have resisted this course of action, for reasons best known to themselves.
The damage that a flawed question can do to a student in what is already a high stakes environment is difficult to overstate. Imagine being faced with a multiple choice question in which all choices are wrong! This will sow self-doubt in all but the most confident and capable student, but all students will waste precious time trying – fruitlessly – to decide where they have gone wrong.
The VCAA claim that students are compensated for errors, but that is not always possible. I know of one student who failed to get the Premier’s Prize for being the top student because the official answer to the question was wrong. The student’s mark was corrected some months later, but no mention of the prize was made.
This situation is not confined to mathematics. In a chemistry exam, students were asked about the property of a peptide, with the molecular structure given. Unfortunately that structure was not that of a peptide.
By employing experts to check the questions (and answers), the VCAA would not only be able to produce examination papers with a much higher standard of reliability but would have someone to blame if a mistake still crept in – much cheaper than hiring a big-4 accounting firm.
Tony Guttmann
Emeritus Professor of Mathematics,
The University of Melbourne
What an extraordinary decision by The Age letters editor.
Perhaps Tony should submit it – maybe in a slightly modified form – as an opinion piece. I believe The Age does publish guest opinion pieces from time to time … Or perhaps he could submit his letter to an alternative masthead.
Did The Age provide an explanation for declining to publish it? Tony can request the letters editor for a reason, if he has the inclination (but who could blame him if he doesn’t).
I don’t think Tony is concerned enough to enquire or to try to publish elsewhere.
Fair enough too. At least the letter has been published here, with a readership of several if not dozens. It’s a black eye for The Age.
To me, the following is significant:
“I know of one student who failed to get the Premier’s Prize for being the top student because the official answer to the question was wrong. The student’s mark was corrected some months later, but no mention of the prize was made.”
This is fact, as opposed to very reasonable speculation that has appeared in the media, regarding the impact of errors.
The readership of this blog over the last couple weeks has not been trivial.
I know. I was making a joke.
Could try the HS (either as opinion piece or letter) noting that the Age has not published it.
Also note how it broadens the potential scope from just maths – that will not be popular with Govt.
Do other states have this problem or is it just VCAA?
Good question. I know little of other states, and haven’t really looked hard, but I think it varies a lot.
I think NSW is basically fine. Extension 2 is way, way stronger than SM, and the questions are good. I don’t know of errors occurring, although I haven’t microscoped. NESA is also way, way more transparent than VCAA: each maths exam is published within a few days, with full solutions in about a month, and exam reports before the start of the new school year.
QLD, the exams are very bad, similar to VCE. Obviously QCAA had the bad and only belatedly caught error last year, and it wasn’t the first. But clearly QCAA takes the issue seriously and, like, NESA, are way more transparent than VCAA.
I suspect that WA is (still) a basket case. In 2018 and 2019 I was alerted to some appalling exam questions. See here, here, here and here.
I know essentially nothing of what goes on in other states.
I’m so jealous of the NSW timeline on releasing their exam materials. No more chasing / sharing scans to see the papers!
As for releasing marking schemes, this should be a requirement. It is an equity thing – not all schools have teachers that can be exam markers or go to the “Meet the examiner” sessions to try to divine how questions are marked.
This applies to all subjects, not just maths – examiner’s reports are just as vague and weaselly in other subjects – and exams full of problematic questions. I was just shown an error in the 2023 VCE Psychology paper where a graph showing % of hours was mislabelled as hours – which made things a bit nonsensical.
There is no (good) reason that VCAA should be worse than NESA in this regard. Where’s our pride?
At the moment, I’m wishing for a pride of lions.
Re: “error in the 2023 VCE Psychology paper where a graph showing % of hours was mislabelled as hours – which made things a bit nonsensical.”
Do you know which question? (One of the MCQ?) I’d stretch a point and call it an out-of-field numeracy error. The sort of error that could appear on a …. NAPLAN.
Sorry BiB, I don’t have a copy. But the question was on the topic of sleep
I can get a copy. I’ll take a look. For efficient looking, do you remember if it was MCQ or short answer?
It’s not that important, but I can’t help wondering how many of these types of questions (the % question in Foundation and this one in Psychology) might appear on a Yr 9 NAPLAN (or a GAT) and contain these sorts of careless mistakes. As we all know, Gniel will (at this stage, anyway) be overseeing the NAPLAN – if he can’t ensure quality in high stakes VCE assessments, one can only wonder what might happen in a lower stakes assessment.
NAPLAN had plenty of clunky questions, and some wrong questions, ten years ago when I was pursuing this. I haven’t looked for a long time, but I see no reason why it would have changed. NAPLAN tests receive even less scrutiny than VCE exams.
I’ve had a look but I can’t see the error being claimed. Which is not to say it’s not there, I might just be blind.
(However, I did learn a little bit about toilet trained cows).
I’m wrong – no need for the letter – it’s not just maths – see https://www.theage.com.au/national/victoria/disappointing-education-minister-orders-inquiry-into-vce-exam-bungles-20231113-p5ejio.html
Thanks, JJ.
I understand this is issue is particularly outside of the domain of this article, forum, or even fight. But, as Guttmann and JJ put above, the erroneous exams extend well beyond mathematics.
Bear with me, but VCAA is much more inclined on retaining the focus particularly on maths exams. It is the main focus, and shining a light on other exams reveals the incompetency so apparent in the examination process. I have heard of many errors and improper questions beyond Mathematics and Chemistry
How poor is the process if not even humanities subjects and art subjects are free? Recall in 2012 Revolutions when artwork from 1917 featured a large mech robot photoshop…
The issue is systemic, and infects every inch of the exam process. The first domino, or hurdle (if better), is the maths exams. VCAA would like for the probing to end there, but it’s an imperative for it to not.
Apologies if I rabbled about irrelevant nonsense.
Indeed. Who could ever forget
2012 History exam (no wonder the Bolshevik’s won!) You’d be hard-pressed to find a more inept display of proof-reading. Read all about it here (under Controversies): https://en.wikipedia.org/wiki/Victorian_Certificate_of_Education
Ken Bruce, you are correct. The trouble is deep-rooted and extends well beyond the mathematics exams. For example, I have heard Geography teachers say that VCE Geography is rife with problems. At the above link, you can read about what happened in the 2022 Geography exam. So what happens in the 2023 Geography exam – the same issue but even worse (this has gone unreported, so far)! Apparently the colour-blind issue was reported to VCAA a few years ago, VCAA listened and made changes to its procedures. But then new people came in and ….
One might think that 2023 has been a particularly bad year for mistakes on VCAA exams. Not so. It’s been like this most other years. The difference this year is that there has been unprecedented scrutiny and interest by both the media and the public. Sleeping giants have woken. I can hear Twisted Sister – We’re Not Gonna Take It. Finally the VCAA cannot sweep their litany of errors under the carpet.
Regarding the Chinese exam bungle, the solution is simple. My understanding is that each year there is a ‘back-up’ paper for all subjects (for if there’s a security breach etc). What should happen is that the back-up paper should be used in next week’s Chinese as a Second Language (SL) exam. And personally, I think that the students who sat the wrong paper last week should re-sit and be given the back-up Chinese Second Language Advanced (SLA) exam (but I doubt this will happen).
Thanks, Ken.
There are two issues with the maths exams:
a) The poor writing;
b) The poor vetting.
Obviously (b) has received the greatest attention recently, but (a) is by the far greater issue. (Of course (a) can lead to (b), especially since the vetters are, similar to the writers, mathematically too weak for the task.)
I do not know how the situation compares for other subjects.
I mentioned this to Marty in private, but would be useful knowledge for others reading: publishing companies (Cambridge, Cengage, etc.) do employ answer checkers for their publications. It’s very basic contractor work where the rate is usually on a per-page basis, I can’t imagine the HR logistics being all that difficult to organise.
It’s not the most glamourous work and quality definitely varies by author, but I’ve definitely picked up some godawful mistakes & misconceptions that would’ve made its way onto this blog if it weren’t for me.
And goodness knows there’s definitely a bunch of mathematicians out there that might appreciate the extra beer money 😉
Thanks, Matt. Of course the more qualified the answer checkers and proofreaders, the better. And perhaps such work makes sense for some junior mathematicians. But, without seeking to discourage people trying, at the stage of checking answers and reading proofs, the horse has already bolted.
When I was a student, the Leaving Certificate was awarded for completion of Year 11. Year 12 was spent trying for “matriculation”, which was university entrance. Examinations for Year 12 were controlled by Russell Love and Tom Cherry, distinguished mathematicians and able teachers.
By the 70s, Year 12 became considered as the natural end of secondary schooling and there was a push for teachers to control the assessment. This was achieved about 1980 and we are living with the consequences.
Indeed we are. I have some references to this sad history in the footnotes of my Do the Maths article. In particular,
Alan Barcan, “Public Schools in Australia from the Late 1970s to the Late 1980s: the Seeds of Change”, Education Research and Perspectives, 37, 2010, 1-37
Have you considered starting a petition to the Victorian Government (https://new.parliament.vic.gov.au/get-involved/petitions/start/)? There’s definitely a lot of frustration currently, about the exam errors, so I’m sure it would get a lot of signatures, especially since people under 18 can sign it (https://new.parliament.vic.gov.au/get-involved/petitions/sign/#who-can-sign-a-petition). If it made a modest demand, such as a parliamentary inquiry into the situation, I can’t imagine the government refusing, although I’m not a politics expert
I think already the Minister has indicated an enquiry (and VCAA a separate enquiry).
It’s fine for people to arrange petitions and the like, and I believe one has been launched, which I’ll write about tomorrow. But it’s not for me to do that.
It has been reported that the Education Minister Ben Carroll said that academics had approached him about the VCE exams and that he has asked “the VCAA to partner with academic institutions in the writing, designing and vetting of exams.”
Source: https://www.theguardian.com/australia-news/2023/nov/13/vce-wrong-exam-victorian-government-chinese-language
Hi,
Gneil was on ABC (Raf Epstein morning show This morning ) saying “sorry” and expressing An “intent” to do better…
But when asked why the exams had not been vetted by qualified mathematicians
He stonewalled only mentioning that
no student would be disadvantaged
Steve R
It was worse than that.
The interview is here:
https://www.abc.net.au/listen/programs/melbourne-mornings/vcaa-chinese-exam-mistakes-review/103102844
I think Gniel did actually claim that the mathematics exams were vetted by qualified mathematicians.
I doubt Gniel or any in the VCAA leadership structure
(https://www.vcaa.vic.edu.au/About-us/our-structure/Pages/Management.aspx)
know the difference between a mathematician and a mathematics educator.
i do not know what the protocols are for those submitting problems on the examinations, but it should include the proposer having to also include complete solutions (along with alternatives that might come up). This will serve three purposes: (1) ascertain that the problem is correct and doable; (2) allow the proposer to pick up errors or lack of clarity in the statement; (2) let the committee better understand the intention of the question and be in a better position to fix any infelicities. Failure to do this warrants an automatic rejection.
Of course, a full-blooded mathematician should *always* assess the questions.
Re: “the proposer having to also include complete solutions (along with alternatives that might come up)”
I would be surprised if this does not already happen. Unfortunately, what the proposer thinks is a complete or correct solution is sometimes very different to what actually is a complete or correct solution. If a question is mathematically defective, there is no doubt in my mind that the ‘solution’ will also be mathematically defective.
Thanks, Ed.
I would assume this happens. But, as BiB is suggesting, the writers and the vetters are simply not up to the job. They are *so* not up to the job, they have no sense of how below the job they are.
Plus, there is, or at least was, some genuine madness in the vetting process. I was told by someone how the infamous “similar block of cheese” came to be. It’s absolutely hilarious, in a black, After Hours kind of way. (And for those about to ask, the answer is “No”.)
But I honestly have no idea of how to thoroughly and securely vet something like a VCE exam. Sure, you read every word, and work through every step. But that is not sufficient to catch all the “what if”s. Some of the latter is simply being able to smell ambiguity and error, and of course a “full blooded mathematician” has a much better sense of smell. But, to switch metaphors, I don’t know how to ensure that you will see what you don’t see.
“VCAA backflips on external audit into VCE exam mistakes” from the Herald Sun – article says Gniel said external review would be better but then changed that 3 days later to internal review by DET.
My surmises – Gniel is out of the loop and review will be overseen by VCAA Chair and DET (as announced by Minister). That way they have more control than outsourcing. And if they outsourced again, they could hardly go tame Deloitte/PwC etc – they would need to get someone very reputable and experienced and likely more independent.
Will very much depend on riding instructions from Minister and skills of people undertaking review (including VCAA Chair). Would guess they’ll set up a DET Steering Committee with VCAA Chair in overall charge. Then DET senior officer and team assigned to do the leg work. These days content skills fairly rare in departments (few doctors in Health Dept etc), so who knows. Lots of options to bring expertise in if they wish to – use sub-consultants for specific tasks, second people in etc.
Thanks, JJ.
It’s all a bit of a fog, but last week the Minister announced there’d be a review (without any specifics, I think). So, it may be less a “backflip” on VCAA’s review than the review being superseded.
Of course you’re right, that the ToR and the people undertaking the review will be key. But a proper review is also going to have to go way, way beyond the “exam writing and vetting processes”. Unless everybody is happy with VCAA continuing to tell falsehoods about 2022.
Until, at least, VCAA comes clean about 2022, the Authority and everybody who speaks for that Authority should be treated with utter contempt.
I cannot believe that despite you and Burkard submitting a detailed critique of the 2022 Specialist Mathematics exams
and despite persistent complaints from teachers
and despite the open letter signed by 70 mathematicians
the VCAA are
denying the existence of serious errors on those exams.
The continued VCAA denial and attempted cover up is unfathomable. The Minister
demand an explanation and that explanation must be made public. The people involved in all this must be sacked.
In politics and the bureaucracy (which is lower level politics), I’m afraid it doesn’t work like that BiB. Just look at Robodebt where it was only a change in Government that brought any real inquiry.
The people in VCAA were the ones who did the Deloitte Review and while it appears they are being moved on, it’s not in the Minister’s interest to have open slather on VCAA (it’s his own Government and also these things can so easily snowball – especially in the modern era).
We don’t know how serious the intention to fix things is. Presuming it is, then it is far better to go for admission of errors when you have a solution to focus on than to dwell too much on them now. It appears the Minister is moving people on. If he is serious, then presumably he will want to understand the problem, have solutions developed etc before acknowledging the scale of the problem.
And never ever ever get your hopes up too much. We used to talk a lot about Karma in the bureaucracy – it was our major hope for the awful people with power.
Thanks, JJ.
If the Minister is moving people on, he’ll need a bus. (And they shouldn’t simply be dropped off at the next bus stop. Their tickets need to be stamped “Never to work in education”).