We would have thought the reality of and the proper administration of mathematics exam errors was pretty straight-forward, at least in broad principle. But, seemingly not. It seems there is a need to hammer a few obvious points:
1. Exams errors are inevitable
They are gonna happen. Of course the prevalence of errors, particularly serious errors, can be significantly lowered by having properly qualified people employed for the writing and the proper vetting, but errors are still going to occur. Exam writing is a very difficult, very human task and, no matter the number of the humans, no matter their intelligence and their diligence, there will always be the potential for oversight.
2. Mathematicians are the proper judges of mathematical error
Properly qualified mathematicians get to determine what is or is not a mathematical error. VCAA apparatchiks do not. Mathematics teachers do not. Maths ed clowns do not. There are judgment calls and anyone has the right to argue their case, but the mathematicians are the judges.
One would have thought this was obvious but it is still not clear that everyone at VCAA (or even at Bennett) has learned this lesson. Certainly not all, or seemingly any, had learned the lesson last year, when “[VCAA’s] maths development team” and “subject matter experts of the exam development process” were quoted denying the undeniable 2022 exam errors. Now, after the mathematicians’ open letter and after Bennett, have VCAA’s “team” finally learned the lesson? We wouldn’t bet on it. No one, from VCAA or anywhere, has been sufficiently reckless to attempt any public defense – we’d love nothing better than for someone give it a go – but anyone too clueless or too arrogant to reflect properly in 2023 is no gimme to do so in 2024.
3. Not all errors have an equivalent effect or are of equal concern
An error can be bad because it is blatant, because it reflects a fundamental misunderstanding of the mathematics, or because it has a detrimental effect on students’ exam performance. An error may be bad in just one way but that oneness does not then excuse the error. Whatever its type(s), an error may be less or more serious.
The estimation of an error requires judgment which, at least in one aspect, does not fall solely to mathematicians. Neither, however, does this judgment fall solely to VCAA and, in particular, it does not fall solely to VCAA’s “psychometric analysts”, whose unexplained voodoo should never be taken on faith.
4. Not all errors should be administered in the same manner
The first and most important responsibility following the discovery an error is how, and whether, to adjust the grading (or regrading). On occasion this adjustment will be obvious: if the question is so garbled as to not be properly answerable then give full marks to all students; if the question has two reasonable interpretations then award properly full marks and sympathetic part marks for correctly done work based upon either interpretation; and so on.
On other occasions, the grading of an error will require a difficult, Solomonesque judgment call. A multiple choice question, for example, may have no correct answer while it still being arguable that the intended answer is preferable; this answer may fail to be true for subtle, “technical” reasons, while it still being plausible that, on average, students who gave the intended answer understood the question better. Should such a question be thrown out? There can be an argument for this, of course, but one can also see why students who thought about and answered the question as intended would be arguing against it. Of course what one person calls “technical” another will call “fundamental”, and that is the point. There is no reasonable black and white approach to handling errors, even on multiple choice questions, and it is foolish to pretend otherwise.
The second and almost as important a responsibility following the discovery of an error is to report the error and its subsequent handling openly and honestly. This raises an interesting conundrum: what does VCAA do when a question containing a clear error should nonetheless, and all things considered, be most reasonably graded as if the error were not present? Should VCAA then write in the report “We goofed but we’ve determined to grade as if we hadn’t”? Yes, they should. One can see why VCAA would be very reluctant to do so, and they needn’t necessarily go full-on with the hair shirt, but it is their responsibility to tell the truth. (At the personal level, if one’s job regularly involves writing or telling lies, including lying by omission, then one might want to reflect upon one’s line of work.)
Of course VCAA has unfailingly failed in respect of both of these responsibilities. For decades. It will be very interesting to see the extent to which Bennett will bring about cultural change on this, if the requirement for VCAA to report in a more detailed and more timely manner will also encourage or necessitate a more empathetic and more transparent approach to the administering and the reporting of errors.
5. Any error, however it is administered, will disadvantage some students
Whatever the method of administering an error, it will be wrong. It will be wrong in the sense that some students will be consequently shortchanged. Even before the grading has commenced, some students will have been shortchanged, by having spent more time pondering the error and/or by becoming more flustered by the error. But also the grading in and of itself cannot give proper credit to all students who deserved it while avoiding giving improper credit to those students who did not. It is impossible, obviously, and no one should ever pretend otherwise. Be honest.
Don’t be a Gniel. One is more than enough.
I agree with all your sensible comments. Still I hate seeing problems like this: “Find the next number in the sequence 2,4,6,8.” I see such problems frequently in on-line tests and professional assessments.
I agree that there is much more to be concerned about in VCE exams, and in exams everywhere, than straight-forward errors. But if the authority cannot deal sensibly with the errors then there’s little hope for the wider concerns.
Well explained. We will see whether the 2024 exams are any better.
Thanks, tom. To be clear, there are two issues for the 2024 exams (and more so for the 2025+ exams). First is the question of whether the exams are “better”: better written, intrinsically better questions, fewer errors. Second is the question of whether VCAA handles the (hopefully fewer) errors in a more reasonable and honest manner.
I am hopeful for some improvement on both fronts, but it’s obviously no gimme.
Did VCAA have an actual policy on how to deal with errors (and do others)? These days there are so many policies on everything, I would be surprised if they didn’t. Probably kept it very vague to provide maximum flexibility, but would be interesting to see how the reviews handled this.
I’m sure VCAA had a policy but, whatever was written, it amounted to usually, if not always, handing the questions to the psychometric analysts to declare that mathematical truth was irrelevant.
Bennett addresses this pretty solidly, around p 33. My memory and a quick scan suggests that Deloitte was entirely useless.
For mine, the VCAA’s denial of point number 5 is the most frustrating.
Rather telling from Bennett in this day and age of policies and procedures being rife (indeed schools are audited on whether they have policies for even eventualities that will not occur as told to me by a Principal of an independent school):
“The VCAA needs to develop procedures for managing issues that have been raised by the public…..”