The most maddening aspect of VCAA is their steadfast refusal to admit error. Typically, VCAA simply will not engage and whatever nonsense they’ve written remains written. 2022 was different, however, in that Burkard and I badgered VCAA sufficiently for VCAA to decide to conduct an external review(s). As well, a mathematics teacher complained strongly enough and unwaveringly enough about the four bad Specialist Mathematics questions that a substantial response to his specific objections was required.
The result so far, however, has been only the continuance of VCAA’s maddening refusal to admit error. Despite a number of reviews, and despite the mathematicians’ open letter, VCAA’s stance is still, at least visibly, that the 2022 VCE mathematics exams “contained no major errors” or “no serious errors that impacted students”, their go-to phrases. The vacuous exam reports remain uncorrected, still just as vacuous, and just as wrong. A few weeks ago, two Department of Education secretaries were able to snow a parliamentary committee with inexcusably ignorant answers, suggesting that the 2022 errors amounted to nothing more than “some language could be tightened”.
How can VCAA maintain this non-sensical, and shameful, position? Obviously, VCAA gamed the reviews to be conducted by mathematical toddlers, to ensure the reviews were as harmless as possible. Still, even mathematical toddlers, when put on the spot, have to babble something, and whatever these toddlers babbled, VCAA has tried their damnedest to keep it hidden. The summary of the reviews that Burkard and I received was laughable in its lack of content. The mathematics teacher, however, fared better.
On 1 December – this 1 December – the mathematics teacher received a letter from VCAA, a response to his insistence that VCAA show their working, which they had previously agreed to do. The letter, which the teacher has shared with us and permitted us to quote, contains “notes” and “analysis” of the four Specialist Mathematics questions. This amounts to VCAA’s best, and only, defence of these questions.
I reproduce VCAA’s entire defence below. I won’t comment much since it has all been said, and nothing needed to be said. You either understand, as do mathematicians, that these questions are “unacceptably flawed”, or you do not. If you do not understand, then you either trust in the public declaration of university mathematicians or you trust in VCAA’s summaries of the opinions of VCAA’s anonymous, hand-picked reviewers.
It should be noted that this is VCAA’s defence from months back, just now being relayed. It is not clear that VCAA still stands by this defence. VCAA has not, however, said anything public to the contrary. Until they do, they continue to own it.
INTRODUCTION AND A GENERAL DEFENCE
VCAA’s letter begins with a general discussion of the external and internal reviews of the exam questions, the “no major mathematical errors” mantra, and so forth. There is not much of substance, but the introduction indicates that the “notes” and the “analysis” provided come from three different sources:
*) “two independent state education authorities”
*) “subject matter experts of the exam development process”
*) “various members of our VCAA team”
It is not always clear which defences can be attributed to which group.
The introduction also semi-defends the multiple choice questions:
The multiple-choice questions have also undergone psychometric analysis which included review of the fit and discrimination of the questions. This analysis found that these questions performed as expected across the student ability range with no anomalies detected.
Similar to VCAA’s treatment of the cheese question, such “psychometric analysis”, of course, indicates nothing about the mathematical correctness or otherwise of the questions. It also indicates nothing of how any one individual deciphered a question, nor how long it took them to decipher it.
Similarly, and similarly off the point, the separate discussions of the four questions contain two almost identical statements, delivering the “performed as expected” message:
“This question was interpreted as intended by the Study Specialist Vetter (SSV) and Examination Sitter Reviewer (ESV).”
“X% of cohort did not respond to the question; Y% scored full marks, discrimination coefficient of Z indicates good/good/fair/very good correlation between student ability and student performance so higher performing students were not likely to be disadvantaged by wording.”
The independent reviewers identified that clarity in the question regarding ‘mean time’ would have assisted students, but identified no mathematical error as such.
What would have “assisted students” is writing what was intended rather than something entirely different.
The case where both a and c are strictly real is a specific case, and may be considered a degenerative case reducing all three roots to real values; a and c are declared to be complex roots and the general case where each has both real and imaginary parts should be considered. Any condition for their imaginary parts to be zero would have been mentioned just as it was declared that Im(b)=0. The question prompts students to work with conjugates in the general case.
The proper term is “generic”, not “general”, as if it matters, since the question “prompts” students to do nothing remotely like what is being claimed. They’re simply making stuff up.
Questions regarding complex polynomials in general form in the subject of Specialist Mathematics must consider the roots of the polynomial to have non-zero real and imaginary components, unless otherwise stated. This is consistent with theory descriptions of conjugate root theorem developed in texts. The constraints in the question did not preclude imaginary elements making option B correct.
Whoever wrote this might consider sticking to integer arithmetic, preferably in the single digits.
The independent reviewers identified that the question was complex, and that it could be confusing to some students due to the ‘necessarily true’ working. Simplification of language for better meaning was suggested.
Did the independent reviewers also suggest the examiners might write what was intended? It seems that “complex” is the new euphemism for “stuffed”.
The question context provided a novel means of testing key knowledge and key skills (specifically: construct approximate confidence intervals for sample means) for students whose learning is at a point where the operational development of the interval using variable values is appropriate.
Describing the question context as “novel” is very apt. A fantasy novel, perhaps? Or Alice in Wonderland?
Independent reviewers indentifed [sic] D as the correct answer and rated the question as complex.
Having now learned the intension of “complex”, we can only agree.
The question involves analysis and requires students to have insight into the physical situation. Students are meant to realise that a machine dispensing varying amounts of fluid will have no connection to, and be independent from, another machine which makes cans of varying mass at the separate sites. Masses of cans will be completed independent of dispensed volumes of liquid.
The question requires more than students “have insight”: it requires that students be able to read minds.
Sure, it’s reasonable to assume that the can masses and liquid masses are independent, although students have never previously been required to make any such independence assumption. But, even if reasonable, this independence assumption is anything but necessary. If, for example, a machine fills a can until a specified total mass is achieved then the amount of fluid dispensed will depend upon the masses of the empty cans, and thus upon the machine that “completed” the cans.
Anyway, let’s take the examiners/reviewers at their arrogant word. So the CAN mass will be independent from the LIQUID mass, and then the TOTAL mass can be treated as the sum of the two independent variables: T = C + L. The variance calculation in the exam report, however, assumes that L = T – C. That is, whereas the letter gives a “well, duh” argument that C and L are independent, the exam report assumes that the C and T are independent.
Navigating this nonsense is like trying to find one’s way in an Escher world.
This was regarded as a complex question by the external reviewers and more information in the question may have assisted students.
Yes, more information, such as the essential information, may well have assisted students. And, yes, the question was indeed “complex”.
Draw your own.