In 2015, pre-blog, I was not the dedicated thug that I am today. I had already taken a few potshots at VCAA, but I hadn’t gone in hard. I understood, of course, that VCAA was prone to being arrogant and inept, but I hadn’t yet concluded that they were systemically arrogant and inept. I hadn’t yet realised the magnitude of the target. So, in 2015, when two teachers approached me complaining about a VCE exam question, I handled it differently than I would now. Then, I was polite and patient with VCAA. We all learn.
The exam question that troubled the teachers was a now infamous multiple choice question on that year’s Further Mathematics Exam 1. The question concerned a block of cheese, pictured above. As indicated, a cut is to be made and the question begins,
A smaller, similar wedge of cheese is cut from the larger wedge of cheese, as shown in the diagram.
This question is undoubtedly familiar to many readers. For all you other guys, we’re guessing it took about a half second to see why the question is mangled beyond repair.
To complete the question, which turns out to matter, the problem was to determine the pictured distance, d, so that the “similar wedge” has half the volume of the original wedge. Cutting the wedge as intended, this leads to d ≈ 2.3 cm: answer B. If, however, students shrank the wedge to be similar, this leads to d ≈ 1.7 cm, which was also an option: answer A.
The appearance of such nonsense on an exam prompts two questions. First, how did such nonsense get approved? Then, secondly, what did VCAA do once the nonsense was called out? I have been told an answer to the first question, but I don’t think I am permitted to tell that story. This post is an answer to the second question.
The exam was held on a Friday, and the following Monday one of the two teachers, let’s call her Joanne, emailed VCAA’s Mathematics Curriculum Manager, politely noting the contradiction between the diagram and the term “similar”. (The other teacher decided not to formally complain because VCAA.) The MCM replied the same day, indicating he had forwarded her email to VCAA’s Examinations Unit.
Two weeks later, because VCAA, Joanne received a reply from the Examinations Unit, from let’s call her Robyn:
Thank you for your email regarding multiple-choice question 9 in the Geometry and trigonometry module of Further Mathematics Examination 1.
When responding to a question, it is expected that students will read the question in its entirety. In Question 9 this includes the diagram of the wedge of cheese in addition to the words in the question. Students are twice directed to look at the diagram. It is clear from the diagram and the wording of the question that only onecut [sic] is made to the wedge of cheese. Given this information, the word ‘similar’ in the question is used in its natural language sense and not in the mathematical sense, as the one cut to the wedge does not change the length dimension of the prism.
VCE Examinations Unit
Unsurprisingly, Joanne was less than thrilled with this response. With no obvious next move, Joanne contacted the MAV, and me and Burkard. The MAV did nothing. I did something, although in the end my intervention probably made no difference to the outcome.
I consulted with a few of my usual consultants, and then emailed the Examinations Unit, with attention to Robyn. This was now three weeks after the exam. After noting the blatant flaw in the question, and arguing against Robyn’s strained defence of the question, I addressed VCAA’s unwillingness to face reality:
… What concerns me as much as the inclusion of such a question is the VCAA’s apparent reluctance to acknowledge failings. From my knowledge of past (in particular Mathematical Methods) exams, this appears to be somewhat of a systemic issue. No one expects the VCAA to handle the very difficult undertaking of VCE exams in a perfect manner. However, I believe we do have the right to expect the VCAA to acknowledge and to address the inevitable imperfections with a significantly greater degree of professionalism, openness and honesty.
A few days later, Robyn replied:
Dear Mr Ross,
Thank you for your email regarding Question 9 of the Geometry and trigonometry module of Further Mathematics Examination 2.
The VCAA is continuing to monitor student responses to this question, and all others. If there is evidence of students being misled by the wording of the question, then we will take the appropriate action to ensure fairness to students.
I responded the next day:
Regards, (Dr.) Marty Ross
(I said I was polite. It doesn’t mean I was a wimp.)
Robyn quickly replied, with the requested email address of her Manager, who turned out to be the Executive Director, Curriculum Division. Let’s call him Dr. David Howes.
That same day, and before I had had a chance to email David, he emailed me, offering to discuss the issue. After a polite “let’s discuss” exchange, David provided a substantial update:
I should first apologise that I did not make clearer to you, through [Robyn]’s response earlier in the week, that we were in the process of analysing all the data from the Further Mathematics exam, and that we would be in contact following the conclusion of that analysis.
It would have been pre-emptive for us to immediately respond to issues that were raised (and that we identified ourselves in the course of the marking process) in relation to Q9 in module 2 prior to having completed the kind of statistical analysis we undertake whenever any issues are identified about exam questions.
We have now completed that analysis. …
Our analysis of student responses to Q9 provides strong evidence that some students may have interpreted the term “similar” using its mathematical meaning rather than its natural language meaning, the latter being the intended usage. This being the case, we are going to accept both options A and B as correct answers for the purpose of awarding the one mark available for this question. In retrospect, the inclusion of the term “similar” clearly did not achieve the objective of making the intent of the question clearer. We will in future be even more rigorous in checking our exams to ensure that, wherever possible terms, the use of any term that may have an ambiguous meaning is avoided.
Thank you for raising this issue with us. I am always keen to engage in constructive discussion that enables us to keep identifying any areas in which we can improve our examinations, so am happy to discuss further any aspect of the above.
In many respects, David’s reply was very good. Most importantly, he indicated that the grading of the question would be adjusted as reasonably as it could be: a mark for either of the two plausible answers. David acknowledged that question was unclear (which was also later acknowledged in the exam report). He was more forthcoming with information than he was required to be, and he was apologetic for Robyn’s previous fobbing reply to me. David seemed genuinely appreciative of the criticism, and his offer to discuss matters further was definitely sincere. (For a couple reasons, I declined.)
For all that, there was something truly maddening, and mad, about David’s reply. Fundamentally, David and/or the Examinations Unit seemed to have no clue how to evaluate the soundness of a mathematics exam question. A question cannot be primarily judged by “statistical analysis” of “student responses”; first and foremost, a question must be judged by the mathematical meaning, or meaninglessness, of the words used. As such, it should have taken VCAA half an hour, rather than the month they did, to conclude that their exam question was stuffed. Nothing in David’s reply indicates any understanding of this.
David also maintained an odd silence about Robyn’s original response to Joanne. That response came two weeks after the exam, and so presumably after VCAA “identified [the issues] ourselves in the course of the marking process”. As such, and anyway, Robyn’s response to Joanne was inexcusable. At some point I indicated to David that Joanne was owed an apology; my memory is that she received one.
This 2015 episode is not all bad; 2022 was way, way worse. But the episode also suggests that there is something systemically flawed in the way VCAA handles exam issues when they inevitably arise. The issue with the cheese question was fundamentally a mathematical issue, with fundamentally a mathematical answer. So why did the Mathematics Curriculum Manager appear to do nothing beyond handballing the question to the Examinations Unit? Did he tell the Examinations Unit that the question was stuffed? Did the Examinations Unit ask him? If so, what did he reply? There are no answers to these questions that reflect well upon VCAA.
Finally, there is The Cheese Story, Part One. As I wrote above, I don’t think I am permitted to tell Part One. I’ll just say, as it was told to me, Part One is black hilarious, much worse than Part Two.