The Bennett Report, 2022 and Mathematics Experts

I would really like to see the back of 2022; VCAA would undoubtedly be thrilled to do so. The continued litigation of 2022, and 2023, is not remotely as important as the current, massive job of repairing VCAA, and the former can be hugely distracting for the latter. (Ill-directed and badly inaccurate news reports are also of no assistance whatsoever.) Nonetheless, there are outstanding questions on VCAA’s honesty and integrity, and there are reasonable concerns that VCAA’s past sources of nonsense may also be current sources of nonsense. I’ll finish with 2022 as soon as possible, but it is not finished until it is finished. Which includes tidying up some loose ends, the purpose of this post.

The Deloitte Report was a farce, generally designed to conclude nothing of substance and specifically designed to avoid any meaningful analysis of the 2022 exam errors. This analysis was undertaken by “NESA” and “QCAA”, but the qualifications and level of expertise of the analysts, as well as the instructions provided to the analysts, are entirely unknown, at least to us. Of the analyses themselves, we know enough and way too little.

The subsequent Bennett Report is different. Like the Deloitte review, the primary focus of the Bennett review was on questions of administration, how to ensure to the extent possible that VCAA would not screw up in the same manner as in 2022 and 2023 (and in every year for at least two decades prior). The Bennett review also paid reasonable attention to 2022 and 2023, however, to consider just how screwed up were the screw-ups. The purpose of this post is to summarise and to briefly comment upon what the Bennett Report contains on the 2022 and 2023 mathematics exam errors. It is much, much better than the Deloitte Report, of course. Still, there are a few things to say.

Two Spoonfuls of Sugar

The Bennett Report has a couple of nice-ish things to say about VCAA. First, on the 2022 and 2023 exam errors (p 7),

The development of Mathematics examinations needs to take into account several factors, amongst them:

• The Study Design specifications

• The technical correctness (or perhaps better, acceptability) of the mathematical phrasing and layout of the questions

• The level of mathematical proficiency that can reasonably be expected from the students.

Balancing these 3 factors is challenging, and the Panel found that in the case of some questions in the 2022 and 2023 VCE Mathematics examinations, mathematical precision and ‘correctness’ have been given a lower priority in the interests of accessibility for test takers.

One can agree with the general premise here if not necessarily, and not likely, the specific conclusion. There is no question that an overly formal or overly pedantic approach to exam writing can create its own problems. There is also no question that it is extremely difficult to conform to VCAA’s appalling and appallingly vague Study Design. The responsibility for which lies with, um, VCAA, but it’s there nonetheless. The claim in the Bennett Report, however, that some of the 2022 and 2023 flubs were simply from a distortion of priorities, from too much concern for “the interests of accessibility for test takers”, requires evidence and is unsupported by evidence.

The writing in VCE mathematics exams is excruciatingly bad, a completely separate problem to that of mathematical incompetence and a massive problem. So, yes, there may be a question or two from 2022 and 2023 that was garbled by reason of a well-intentioned if ill-conceived trade-off reasons, but any suggestion that VCAA’s exam screw-ups are in decent part due to such trade-offs is laughable.

Secondly, on VCAA’s access to mathematicians (p 22),

The VCAA has positive, professional relationships with some Mathematics academics who provide sound and constructive support to a variety of projects, including the development of VCE examinations.

To which the only appropriate response is, “Bullshit”. These relationships may be “positive” and “professional”, and the academics may “provide sound and constructive support”, but the end product is what it is. The end product is garbage.

These passages in the Bennett Report rankle but they should not be over-interpreted. The passages might be considered a bit of sugar, added to make more palatable the large doses of medicine that the Bennett Report administers. The report is written in a diplomatic, and arguably over-diplomatic, manner, but the report is also strong. The messages are clear.

Bennett’s Mathematics Experts

The Bennett Report is a little more informative than Deloitte on the people engaged to analyse the past exam questions (pp 5-6):

The Panel also engaged 7 independent Mathematics experts including university academics from interstate, past chairs of VCE Mathematics setting panels, and professional educational assessment experts specialising in Mathematics.

Elsewhere (p 14), the report refers to

… 7 highly qualified experts, including Mathematics academics from interstate universities and other highly credible educational organisations, and other Mathematics assessment experts.

There a few things to say about this, the first of which is that Dr. Bennett, good writer though he is, should learn to refer to cardinal numbers with names rather than numerals: the more elegant phrasing is “seven highly qualified experts”. Of slightly more concern, however, is the meaning of “highly qualified experts”.

To begin, the phrase “university academics” is remarkably vague. Were these professors of history, perchance? Probably not, but could they have been academics in mathematics education? Which is, of course, exactly the kind of sleight of hand that is endemic in these disputes, with maths ed clowns routinely paraded as “mathematics experts”. Presumably not, presumably “mathematics academics” means what it says. Except, …

Along with the university academics we have “past chairs of VCE Mathematics setting panels”. These are “Mathematics experts”? Well, maybe “past chairs” means long past, that Bennett dug up Professor E. R. Love and reanimated his corpse. Failing that, the description “mathematics experts” would appear to be a hell of a stretch.

Finally, we have “professional educational assessment experts specialising in Mathematics” and “mathematics academics from … other highly credible educational organisations”. Which sounds like a really really clumsy way of spelling “ACER”. Who wouldn’t know a mathematics expert if it bit them.

None of this is clear or encouraging, and there seem little reason for people to trust that Bennett’s “7 independent Mathematics experts” possessed a uniformly strong mathematical expertise. Still, perhaps naively, I am not totally dismissive.

I met John Bennett and his panel, as part of the review, and I was impressed. Bennett is a smart, no-nonsense person, and he also gave the impression of being no mathematical slouch: not remotely an expert, but Bennett was a mathematics teacher in a former life and he seemed at least somewhat aware that “Mathematics expert” means something. Bennett was also CEO of NESA for a long period, and he would very likely still have personal access to solid mathematicians. The description in the Report of the “7 highly qualified experts” is close to meaningless, and there is good reason to doubt that there was a consistently high level of expertise, but my guess is that there was more solidity there than is indicated by the report. In any case, these “experts” saw what they saw.

The 2022 Errors

Unsurprisingly, the basis of the Bennett review’s consideration of the 2022 exam questions was Burkard’s and my critique (p 16). Along with the five major errors, our critique singled out many other exam questions, for being poorly worded or poorly conceived, and so forth. The review considered all these flagged questions, in a careful and intelligent manner, as documented in Appendix 3 of the Report (pp 41-43). In particular (p 42),

A sample of questions was selected for further inspection. The selection made was based on the severity of the criticism, and the degree of consensus among the 7 experts as to the quality of the questions. For example, the 5 questions in the 2022 Specialist Mathematics and Mathematical Methods examinations that had been claimed to contain ‘Major Mathematical Errors’ were examined in detail. Where there was clear disagreement with the claims, or substantial difference in the judgements among the experts, it was deemed that the questions were of adequate quality.

So, all five of the “major error” questions made the cut for further inspection, seemingly on the basis of Burkard’s and my hammering, if nothing else. Other questions, however, on which the experts substantially disagreed, were deemed to be “adequate”. To which there are two responses. First, it should be noted that “adequate” is well short of “good”. Secondly, one must consider the possibility that some of these questions were not (even) adequate and, rather, some of the “experts” were inadequate. Without the analyses, we cannot tell. In any case, there was seemingly little disagreement on the five major questions.

All of the experts considered the Specialist Mathematics questions, and all but one (due to shortage of time) considered the Mathematical Methods questions (p 42). In summary (p 7, 16),

Regarding the 2022 examinations, concerns had been expressed by a significant number of Mathematics academics around 5 questions in particular, which they described as ‘unacceptably flawed’. These 5 questions were also identified as containing errors by all or all but one of the independent Mathematics experts engaged by the Panel.

Most or all of the 7 Mathematics experts identified the same errors in 5 questions in the 2022 Mathematics examinations that had been raised in the original open letter signed by 73 Mathematics academics.

So, six out of seven (and possibly six out of six). It would be helpful to have the analyses but the vote is enough: the Bennett Report came to the clear conclusion that the major errors were major enough errors. Which of course they were.

The 2023 Errors

The 2023 exam questions were less thoughtfully considered by the Bennett Panel because this consideration was based upon a less thoughtfully produced critique (p 16). Which was provided by me. At the time, I had zero time and negative energy to produce a proper critique, but I guessed that the Panel would want something, and of course no one else was going to do it. So, I quickly cobbled together a list of “significant errors”, based upon this post. This list was introduced with an explanation and disclaimer:

The following is the list of the twenty-two significant errors on the 2023 VCE mathematics exams of which I am currently aware. There may well be more: I have not looked at the General Mathematics and Foundation Mathematics exams beyond the questions flagged to me, and I have not “vetted” the other exams, or anything close to it. By “significant” error, I mean a question that: is sufficiently ambiguous or improperly written to have been actively confusing to students; or that cannot be properly answered; or that exhibits some fundamental mathematical misunderstanding. The exams also contain a number of other, minor errors. And, as in 2022 and in every year, there is much more about the exams that is open to serious criticism.

The list was monotone and clunky, it did not give proper emphasis to the extended debacle that was Foundation Mathematics, and it missed a pretty major error on Methods Exam 2 (MCQ 18). Nonetheless, the list sufficed. Due to time constraints, the reviewers ignored Foundation Mathematics. On the other exams (p 15),

• Two of the Mathematics experts engaged by the Panel considered the 5 General Mathematics questions. They found that these questions were poorly worded and could have been improved. Two questions had more than one possible answer and another had an error in the question.

• The 14 questions alleged to have major or significant errors in Mathematical Methods and Specialist Mathematics were each considered by 6 or 7 experts.

• In 9 of those questions at least 5 or 6 of the experts found that the questions were poorly worded or used poor terminology. Comments such as these were offered: ‘The question is very badly worded leading to different possible interpretations.’; ‘This is another example of a lack of precise language in a question.’; ‘Not well phrased …’. In some cases, this wording was regarded as likely to have caused confusion among some students. In other cases, the consensus was that, though the terminology was not strictly correct, it was more a matter of poor wording or expression than a substantive flaw: ‘a minor wording issue. The intent of the question was clear’ and ‘Technically, you rotate the region between the curve and the x-axis to form the solid, but this is clear from the context.’ In still other cases, the experts found the wording of the questions unnecessarily verbose or clumsy: ‘Introduction contains the information necessary for both parts but is overwhelmingly wordy’; ‘there is too much reading’.

• In 2 questions the experts questioned whether the term ‘smooth’ – as used in Mathematical Methods Exam 2 (MCQ9) and Specialist Mathematics Exam 2 (Q1b) – was within the Study Design.

• Three of the 14 questions were found to be acceptable by all 7 experts.

So, there was broad but not universal agreement from the reviewers with the provided error list. Which is fine. As I indicated, the list was hurriedly compiled and, with hindsight, I might agree to downgrade the severity of some of the criticism. I might now agree that some of the exam questions are “acceptable”. But probably not.

Judging by the selected quotes above, the reviewers’ minor pushback was concerned with language: where I had argued that some wording amounted to a significant error, the reviewers’ concluded that “it was more a matter of poor wording or expression than a substantive flaw”, and that “The intent of the question was clear”. One might normally have some sympathy for this point of view, but in this case there are two strong reasons to take a much tougher stance.

The first issue is that dismissing concerns of poor wording with a flippant “intent was clear” ignores the likelihood that such poor wording will come back to bite. Which it did. So, yes, students would have interpreted the “solid” formed by rotating “the curve” in the only reasonable manner, as rotating the underlying region. Which is semi-vaguely fine. Except, because VCAA had routinely screwed up the terminology, when VCAA actually wanted to rotate a curve to create a surface they over-corrected and came up with the idiotic phrase, “open hollow surface”.

The second and much greater issue is that the reviewers’ conciliatory stance fails to take into account VCAA’s long and proud history of being a gang of self-righteous and anal retentive assholes. There does not exist an exam report that fails to whine about some triviality of wording or presentation. If one then dared to approach VCAA, to claim “the intent of the answer was clear”, the response was invariably “Tough titty”. One should typically offer some human leeway on such failings but VCAA has earned absolutely none. They made their nasty, pedantic bed and now they must lie in it.

Conclusion

The Bennett Report has issues. The blasé use of the phrase “mathematics experts” is unfortunate and probably cloaks at least some lack of expertise. In places, the Report is probably too gentle and too forgiving. But, by large, they got it right. They concluded, correctly, that VCAA screwed up. Royally.