A few years ago, before having introduced the secret exam business and the exam error lists, I posted a WitCH on a question from the previous year’s Specialist Exam 1. For me, there was a glaring issue with the question. It turned out that the commenters, while noting other clunky aspects, could not see this issue. Now my commenters includes, of course, the smartest maths teachers around, and so I was puzzled.1 So much so, that I checked the question with some mathematicians. As I wrote in a comment,
Given that commenters are not seeing what I see as a central flaw in the exam question, I asked five people – four mathematicians and one (very smart) non-mathematician – to take a *one-minute* look at the exam question, and to tell me what they saw. All four mathematicians noted the flaw (plus some of the other nonsense), and the non-mathematician did not.
This for me was a very telling episode. The exam question can be done in an autopilot manner to give the intended answer, but what the question was literally asking was entirely different, permitting infinitely many and fundamentally different answers. Mathematicians very quickly saw the question in this literal (i.e. correct) sense. True, there was little likelihood of any students except perhaps the very brightest veering off the intended track,2 but this did not save the question. As I wrote when updating the WitCH,
Whatever the immediate practical concerns, however, mathematicians are aghast at this error. They are aghast because the exam question is simply not testing mathematics. Yes, the students went through the ritual and attempted to compute what was intended and were graded accordingly. And, yes, teachers can now coach current and future students on the required ritual. But none of that is mathematics and, indeed, it is worse: it is antimathematics. It is teaching students to ignore mathematical meaning, to see no value in mathematical precision, to respect only ritual.
This is the difference between exams being under the control of mathematics teachers, even smart mathematics teachers, and mathematicians. A mathematician3 will immediately see the flaw in the exam question and will recognise that the question thus written is fundamentally unacceptable; a mathematics teacher, even a smart mathematics teacher, typically will not and will not. As I keep hammering, mathematicians are the proper judges of mathematical error. Which, like Nicolas Cage returning to Red Rock West, brings us back to the 2022 exams.
A couple months ago, I wrote a bit on VCAA’s internal response to Burkard’s and my critique of the 2022 exams. As I indicated at the time, this internal response (obtained via FOI) is long and absurd, so much so that it is difficult to know how to address it. I have thoughts for a few posts in the next few days, and below is a small and easy bit, with the same message as the WitCH discussed above.
As the centrepiece of our critique, Burkard and I indicated five major errors on the 2022 exams. This included an error in a multipart question on Methods Exam 2. That question defined a “region” in terms of a parameter “k > 0”, and then defined the function A(k) to be the area of the region so defined. Later, the question instructed,
Explain why the domain of A(k) does not include all values of k.
The intent was that values of k for which the region was empty/non-existent be excluded from the domain of the function A(k), but there is no reason to do so and there are multiple other valid ways to “explain” what is going on. As Burkard and I explained in our critique,
The question is expressed badly, and is effectively meaningless for multiple reasons. First, it has been declared at the outset that k > 0. Second, the question has introduced the function A(k) implicitly for all positive k as if this is not a concern, and before part (i): any querying of the domain should have been done immediately. Third, the question in part (i) only makes sense if one accepts that A(k) = 0 also makes sense, implying that all such (positive) k would be in the domain of A, irrespective of whether the region is empty. Fourthly, the conclusion of the question is best though to be false: there is no reason why one cannot consider a region to be empty, and consequently A(k) make sense for all (positive) k.
It was a very stupid question, very stupidly worded.
Following Deloitte, and after VCAA CEO Stephen Gniel and his henchmen finally got around to telling me and Burkard, and everyone, to bugger off, Burkard arranged for the open letter, to be signed by mathematicians and subsequently sent to Victoria’s Minister for Education. For this, we had to decide precisely what evidence to send to the mathematicians and precisely what they were to sign on to, to make it as transparent and as simple as possible. We agreed to just present the major errors, together with the (finally available) exam report (non)comments, but Burkard argued that we should leave out the Methods error. He agreed it was a major error but was concerned that VCAA, and perhaps even some mathematicians, simply wouldn’t get it. I insisted the Methods error be included, mostly for strategic reasons: since we had included the Methods error as a major error in our original critique, I felt we had an obligation to defend our original (and correct) stance.4
Burkard had a point: the issue with the Methods question is more subtle than with the other 2022 questions, and non-mathematicians are much less likely to get it.5 The 2022 VCAA clowns didn’t get it, even when their noses were shoved in it.
VCAA’s internal response to Burkard’s and my critique of the Methods question comprised the view of one, unidentified, person. (It is unclear why the views of other persons involved were not obtained and/or included.) Here is their view:
The question was devised to prompt students into utilizing their key knowledge and skills to examine parameters that would permit the area as shown in the diagram (specified in the question) to exist. The question also causes students to necessarily engage their CAS in a deliberate manner. The solutions rewarded students who could either utilize CAS or could corretly [sic] identify the differential properties at the graphs [sic] origin (asked for in part b of the same question.
All entirely irrelevant. It is evident that this person either failed to understand the flaw in the Methods question or was simply unbothered. Seemingly both. This person, who is obviously not remotely a mathematician, played some key role in 2022; they should be permitted no similar role in 2025.
One of the key recommendations of the Bennett Review was for VCAA to employ properly solid mathematicians in the examination production process. Before being shafted by populist thugs, VCAA CEO Kylie White was doing just that. One dearly hopes that this has continued. One dearly hopes VCAA understands that the nitwits who were in charge of the 2022 exams are not remotely mathematicians, that this matters and that these nitwits are not to be trusted with matters of mathematical truth.
1) Yeah, I’m half-pandering and half-joking. I am generally impressed, however, by the work and sense teacher-commenters put into the questions and issues raised on this blog. In this instance, they came up short.
2) Students performed poorly on the question, but for reasons other than the existence of the fundamental flaw.
3) One has to be careful with the definition of “mathematician”. Australia is a village, with no shortage of village idiots. I know “mathematicians”, even of professorial rank, whom I wouldn’t trust with a pair of scissors much less a VCE exam.
4) I also had a personal reason for keeping the Methods question in. The day after the Methods exam I met with a crazy-smart girl who had done the exam (years ahead of schedule). I asked her how it went and the very first thing she noted was the question in error: she had no idea what was expected, had answered “Because you already said k > 0”, and thus would have been screwed over on that question. It was the archetypal case of VCAA’s stupidity resulting in the screwing over of a smart student.
5) Deloitte’s NESA guy, whoever they are, picked up the Specialist Mathematics exam errors but not the Methods error. (Deloitte’s QCAA guys picked up nothing about anything.) The Bennett Review “experts” picked up all five major errors, including the Methods error.