VCAA’s Defence of the Indefensible 2022 Exam Questions

The most maddening aspect of VCAA is their steadfast refusal to admit error. Typically, VCAA simply will not engage and whatever nonsense they’ve written remains written. 2022 was different, however, in that Burkard and I badgered VCAA sufficiently for VCAA to decide to conduct an external review(s). As well, a mathematics teacher complained strongly enough and unwaveringly enough about the four bad Specialist Mathematics questions that a substantial response to his specific objections was required.

The result so far, however, has been only the continuance of VCAA’s maddening refusal to admit error. Despite a number of reviews, and despite the mathematicians’ open letter, VCAA’s stance is still, at least visibly, that the 2022 VCE mathematics exams “contained no major errors” or “no serious errors that impacted students”, their go-to phrases. The vacuous exam reports remain uncorrected, still just as vacuous, and just as wrong.  A few weeks ago, two Department of Education secretaries were able to snow a parliamentary committee with inexcusably ignorant answers, suggesting that the 2022 errors amounted to nothing more than “some language could be tightened”.

How can VCAA maintain this non-sensical, and shameful, position? Obviously, VCAA gamed the reviews to be conducted by mathematical toddlers, to ensure the reviews were as harmless as possible. Still, even mathematical toddlers, when put on the spot, have to babble something, and whatever these toddlers babbled, VCAA has tried their damnedest to keep it hidden. The summary of the reviews that Burkard and I received was laughable in its lack of content. The mathematics teacher, however, fared better.

On 1 December – this 1 December – the mathematics teacher received a letter from VCAA, a response to his insistence that VCAA show their working, which they had previously agreed to do. The letter, which the teacher has shared with us and permitted us to quote, contains “notes” and “analysis” of the four Specialist Mathematics questions. This amounts to VCAA’s best, and only, defence of these questions.

I reproduce VCAA’s entire defence below. I won’t comment much since it has all been said, and nothing needed to be said. You either understand, as do mathematicians, that these questions are “unacceptably flawed”, or you do not. If you do not understand, then you either trust in the public declaration of university mathematicians or you trust in VCAA’s summaries of the opinions of VCAA’s anonymous, hand-picked reviewers.

It should be noted that this is VCAA’s defence from months back, just now being relayed. It is not clear that VCAA still stands by this defence. VCAA has not, however, said anything public to the contrary. Until they do, they continue to own it.

 

INTRODUCTION AND A GENERAL DEFENCE

VCAA’s letter begins with a general discussion of the external and internal reviews of the exam questions, the “no major mathematical errors” mantra, and so forth. There is not much of substance, but the introduction indicates that the “notes” and the “analysis” provided come from three different sources:

*) “two independent state education authorities”

*) “subject matter experts of the exam development process”

*) “various members of our VCAA team”

It is not always clear which defences can be attributed to which group.

The introduction also semi-defends the multiple choice questions:

The multiple-choice questions have also undergone psychometric analysis which included review of the fit and discrimination of the questions. This analysis found that these questions performed as expected across the student ability range with no anomalies detected.

Similar to VCAA’s treatment of the cheese question, such “psychometric analysis”, of course, indicates nothing about the mathematical correctness or otherwise of the questions. It also indicates nothing of how any one individual deciphered a question, nor how long it took them to decipher it.

Similarly, and similarly off the point, the separate discussions of the four questions contain two almost identical statements, delivering the “performed as expected” message:

“This question was interpreted as intended by the Study Specialist Vetter (SSV) and Examination Sitter Reviewer (ESV).”

“X% of cohort did not respond to the question; Y% scored full marks, discrimination coefficient of Z indicates good/good/fair/very good correlation between student ability and student performance so higher performing students were not likely to be disadvantaged by wording.”

 

Specialist Mathematics Exam 1, Q3(b) (The coffee machine: see here, here, here and here)

The independent reviewers identified that clarity in the question regarding ‘mean time’ would have assisted students, but identified no mathematical error as such.

What would have “assisted students” is writing what was intended rather than something entirely different.

 

Specialist Mathematics Exam 2, MCQ4 (Complex numbers: see here, here, here and here)

The case where both a and c are strictly real is a specific case, and may be considered a degenerative case reducing all three roots to real values; a and c are declared to be complex roots and the general case where each has both real and imaginary parts should be considered. Any condition for their imaginary parts to be zero would have been mentioned just as it was declared that Im(b)=0. The question prompts students to work with conjugates in the general case.

The proper term is “generic”, not “general”, as if it matters, since the question “prompts” students to do nothing remotely like what is being claimed. They’re simply making stuff up.

Questions regarding complex polynomials in general form in the subject of Specialist Mathematics must consider the roots of the polynomial to have non-zero real and imaginary components, unless otherwise stated. This is consistent with theory descriptions of conjugate root theorem developed in texts. The constraints in the question did not preclude imaginary elements making option B correct.

Whoever wrote this might consider sticking to integer arithmetic, preferably in the single digits.

The independent reviewers identified that the question was complex, and that it could be confusing to some students due to the ‘necessarily true’ working. Simplification of language for better meaning was suggested.

Did the independent reviewers also suggest the examiners might write what was intended? It seems that “complex” is the new euphemism for “stuffed”.

 

Specialist Mathematics Exam 2, MCQ19 (Confidence intervals: see here, here, here and here)

The question context provided a novel means of testing key knowledge and key skills (specifically: construct approximate confidence intervals for sample means) for students whose learning is at a point where the operational development of the interval using variable values is appropriate.

Describing the question context as “novel” is very apt. A fantasy novel, perhaps? Or Alice in Wonderland?

Independent reviewers indentifed [sic] D as the correct answer and rated the question as complex.

Having now learned the intension of “complex”, we can only agree.

 

Specialist Mathematics Exam 2, QB(6)(f) (Cans of Liquid: see here, here, here and here)

The question involves analysis and requires students to have insight into the physical situation. Students are meant to realise that a machine dispensing varying amounts of fluid will have no connection to, and be independent from, another machine which makes cans of varying mass at the separate sites. Masses of cans will be completed independent of dispensed volumes of liquid.

The question requires more than students “have insight”: it requires that students be able to read minds.

Sure, it’s reasonable to assume that the can masses and liquid masses are independent, although students have never previously been required to make any such independence assumption. But, even if reasonable, this independence assumption is anything but necessary. If, for example, a machine fills a can until a specified total mass is achieved then the amount of fluid dispensed will depend upon the masses of the empty cans, and thus upon the machine that “completed” the cans.

Anyway, let’s take the examiners/reviewers at their arrogant word. So the CAN mass will be independent from the LIQUID mass, and then the TOTAL mass can be treated as the sum of the two independent variables: T = C + L. The variance calculation in the exam report, however, assumes that L = T – C. That is, whereas the letter gives a “well, duh” argument that C and L are independent, the exam report assumes that the C and T are independent.

Navigating this nonsense is like trying to find one’s way in an Escher world.

This was regarded as a complex question by the external reviewers and more information in the question may have assisted students.

Yes, more information, such as the essential information, may well have assisted students. And, yes, the question was indeed “complex”.

 

CONCLUSIONS

Draw your own.

 

18 Replies to “VCAA’s Defence of the Indefensible 2022 Exam Questions”

  1. Regarding Specialist Mathematics Exam 2, QB(6)(f) (Cans of Liquid):

    I’d like to add that the two different assumptions of independence lead to two different calculations which in turn lead to two different answers (the fact that both answers are the same if rounded to three decimal places is beside the point) – see attached. A clear statement in the question as to which random variables are to be treated as independent is therefore essential, despite assertions to the contrary.

    I would also like to make it clear that two different assumptions of independence leading to two different calculations means that there are two different ways of earning full marks (once we’ve found our way in an Escher world). However, both the Exam Report and the VCAA Defence suggest that only ONE of those ways was acceptable (and they contradict each other as to which one).

    2022 QB(6)(f) (Cans of Liquid)

    1. I think most readers of this blog are familiar enough with the details of the questions. Or, they can follow the links.

  2. Escher? Surely not. Escher may lead one astray, but does so in a *structured* way. This looks more like Kafka (where there is no way to make sense of how one might elicit a useful response).

    I assume the “unwavering teacher” is John Kermond. He deserves our thanks – and a medal! His fate (see his second submitted document, page 2 of 4: “Tuesday 14 February, 3:40 pm: Meeting with my Principal. Five allegations raised against me by VCAA” etc.) may explain why many submissions are anonymous.

    One of the great paradoxes of modern western liberal democracy emerges when one observes (i) the behaviour of western bureaucracies in comparison with those of certain regimes we tend to “look down on”; (ii) the frequent absence in the comfortable west of professional engagement and political courage in the face of bureaucratic nonsense, as compared with what one finds in more turbulent and violent times and places. As examples I offer: (i) the Soviet Union in the 1930s, and the curious cameo involving Leonid Leonidovich Mischchenko, that forms the central focus of https://arxiv.org/pdf/2105.10979.pdf ; (ii) your own recent experience with “10 boys and 10 girls” (it is worth googling “10 boys and 10 girls” just to see how truly unproblematic this phrase seems to be).

    1. Fascinating article. Thanks for sharing, Tony. Loren Graham has an excellent book on the history of Moscow’s mathematical school called ‘Naming the Infinity’. The pdf of the book could be easily found using Google search. It tells the story of the communist oppression of mathematics and mathematicians in the Soviet Union.

    2. Thanks very much, Tony. Generally, it’s definitely Kafka. But the can question seems like Escher, in that it keeps looping back.

      Re the “unwavering teacher”, yes he deserves a hell of a lot of credit. And many organisations deserve zero credit, or less: the Heads of Maths group did nothing; AMSI did nothing; AustMS did nothing; AAMT did nothing; MAV did worse than nothing. To hell with them all.

      The Mishchenko story is wild. Thanks.

      Re the “ten girls and ten boys” AMT nonsense, that is continuing. I plan to write something in the next day or so.

      Finally, one shouldn’t forget the crazy Unbowed story.

  3. Shambles, so much effort instead of simply admitting the error and explaining what they will do to avoid repeating it.

  4. It is all nutty, but reading their defence of Specialist 2, MCQ4 takes the cake. That “argument” is like insanity in written form. It’s either that, or an intense misunderstanding of complex numbers and a deliberate choice to not understand them. It seems as though they regard purely imaginary or purely real numbers as not being complex, and therefore should be discounted somehow?

    Another mad aspect is that multiple groups of people contributed to this thing. How embarrassing.

    1. Thanks, Glen.

      Yes, the level of insanity of the complex question, and now its defence, is difficult to comprehend. As you note, that “multiple groups of people” could seemingly read the question and review the question, and do anything other than vomit condemnation is astonishing. But there’s a lot we don’t know.

      To begin, we don’t know how many people were involved in the writing and the vetting and the reviewing of the question. Asked “how many sets of eyes would see a final exam”, David Howes simply replied, “many”. (To be fair, Howes wasn’t asked how many brains would consider a final exam.) But Howes’ comment also makes puzzling the letter’s remark about “the” Study Specialist Vetter and “the” Examination Sitter Reviewer”. Why the definite articles?

      Then, reading the excruciatingly worded question, it seems simply that the writers stuffed up, that they had intended to declare Im(b) ≠ 0 rather than Im(b) = 0. But if so, when and why did the insane “general case” argument appear, and who came up with the argument?

      It feels to me as if the “general case” argument is post hoc. But, if so, whose argument is it? I’d really like to know which of the three groups is responsible, and what the other two groups really said. Note that we don’t know what any of these groups said: we only have the letter’s summary, and there are obvious reasons to have little faith in this summary, particularly in regard to any negative comments that anyone might have made.

      1. Just to clarify, the three groups you refer to are

        *) “two independent state education authorities”

        *) “subject matter experts of the exam development process”

        *) “various members of our VCAA team”

        We not only don’t know what any of the groups said, we don’t even know the membership of each group or the qualifications of the membership.

        We might assume “the” Study Specialist Vetter and “the” Examination Sitter Reviewer” are part of the second group, but even that’s not certain. What seems apparent is that none of the groups contained an attentive and competent mathematician. Which is at odds with what VCAA has regularly said. We would learn a lot if VCAA simply stated the qualifications of their mathematician(s) (no names required) because I think VCAA has Mathematician agnosia.

        Re: The “Many” answer. In fairness, that’s how the Pirahã tribe count.

      2. I wrote this wrong: the natural guess I was suggesting was that the writers intended to write Im(a) ≠ 0 and Im(c) ≠ 0, rather than Im(b) ≠ 0 (which wouldn’t work). But I now have other thoughts as well. I’ll post soon.

    2. They don’t believe what they’re saying. If they did, then the original question wouldn’t have specified that “Re(a)≠0, Re(b)≠0, Re(c)≠0”

      1. Thanks, Jay. I imagine they think they can weasel out of it, by playing their “unless otherwise stated” card.

        EDIT: No, I think you’re right. Given their stated stance, there is no purpose to writing a real or imaginary part is non-zero, only that a real or imaginary part is zero. So, their argument is screwed (as if it wasn’t already screwed).

  5. The more I think about this question, the more frustrated I get (and I’m sure others are even more so), but this line:

    “Questions regarding complex polynomials in general form in the subject of Specialist Mathematics must consider the roots of the polynomial to have non-zero real and imaginary components, unless otherwise stated.”

    Is just too much.

  6. I think I know what happened. VCAA got Bud Abbott and Lou Costello (*) to review those questions. It all makes sense now.

    * For the many younger readers: No, they are not ex-politicians. Abbott and Costello were an American comedy duo of the 1940’s and 1950’s. I’m referring to their famous 7×13=28 skit.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here