VCAA’s Response to the 2022 Critique

This post will shed a little light on VCAA’s inner sanctum; not much, but a little. First, the motivation for this post.

Kylie White has gone. Perhaps that was always going to be the case, terrible decision though it is, but the media crusade against her undoubtedly helped. The last arrow fired in this crusade, the day before White resigned, was, predictably, an article by the Herald Sun‘s Susie O’Brien:

VCAA staff dragged into exam scandal as authority accepts fault

You gotta love the Herald Sun‘s headlines. In this case, a more accurate headline would have been,

Susie O’Brien managed to drag VCAA staff into exam scandal

Manipulative tearjerker headline notwithstanding, O’Brien had a genuine story: a VCAA “insider” had provided O’Brien some details of the chaos inside VCAA as they scrambled to repair the exams. Fair enough. One can query the motivations of the “insider”, which are relevant to evaluating the story, but these motivations are not properly O’Brien’s concern. There was one quote from the “insider”, however, that stood out:

“There is one (person) who has almost killed herself trying to fix [the exams] whilst high-level leaders were going home.”

That is an incredibly scummy line.

We are not told which “high-level leaders” were “going home”, but we’re conveniently led to imagine it included White, who was waltzing away at knock-off time, queenishly leaving her underlings to clean up the mess. I do not believe anything of the sort occurred. No one should.

One can argue that the buck stops with White and what that implies. One can argue that the exam mess was handled poorly once discovered. (I have no idea.) But there is zero reason to doubt that everyone in VCAA was working frantically to fix the mess once it was discovered.

That nasty backstabbing line is the reason I’m writing this post, or at least the reason I’m writing it now. I cannot shed much light on VCAA’s inner sanctum but I can shed a little, and that one nasty line makes it worth doing.

*******************************************************************

Some time after the 2023 exam mess, I submitted two FOIs to VCAA. The FOIs were centred around VCAA’s sham Deloitte review of the 2022 exam mess but were quite broad. So, the (redacted) Deloitte review was released but so also were other documents, and the process is on-going. (FOI processes are always long and frustrating, and this one is no exception, but I would like to make clear that it has also been amicable. While I disagree with and am appealing aspects of VCAA’s decisions, VCAA’s FOI officer has treated me respectfully throughout (and I believe I have reciprocated).) Most of the documents released have turned out to be of little interest but there are exceptions. This post is about an exception.

In preparation for the external review to come, VCAA requested that Burkard Polster and I put in writing all our concerns regarding the 2022 exams. We did so and the result was our critique (embedded below). As it turned out, this was farcical: Deloitte did absolutely nothing with the critique. Our critique was used to select the questions that NESA were to review (Appendix 4), but there was no external analysis whatsoever of the critique. There was, however, an internal analysis.

One of the documents released in the FOI is an untitled VCAA document, a “Specific responses” item-by-item rebuttal of Burkard’s and my critique (SR). There is no indication of the author/compiler of SR; the push-backs on specific exam questions are from various, semi-specified sources, including (unnamed) exam writers, exam reviewers and people internal to VCAA. There is also no indication when SR was composed, or why. Presumably it was written in early 2023, in the early Deloitte days, but there is nothing in the Deloitte report, or in anything I’ve seen, that indicates how SR might have been used or by whom.

SR is long, and it is astonishing, the height of its arrogance and the depth of its delusions difficult to fathom. SR is also overwhelming: Burkard’s and my critique was long and detailed, and SR is long in response, and tiring. It is a kind of reverse Gish gallop: rather than simply admit that they screwed up, VCAA provided nonsense arguments at great length.

Here, I will only reproduce and comment upon a small portion of SR. The large body of SR consists of comments on specific exam questions, amounting to a rebuttal of Sections 2-5 of Burkard’s and my critique. (SR wrongly indicates that this body is responding to Section 6 as well, but SR nowhere contains a response to this small section.) There is plenty of “you gotta be kidding” material in the body, some of it walloped previously, but there is no clear purpose in reproducing it as a whole. I’m pondering what to do further with SR, but for now there is a shorter and clearer way to give a sense of the thing.

The first part of SR is a list of twelve dot points, VCAA’s response to the critique’s preamble and Introduction (S 1), Delay and Dishonesty in the Exam Reports (S 7), and Conclusion (S 8). I reproduce the dot points below, together with some comments. Then, a brief and very depressing conclusion.

*******************************************************************

1. The authors’ concerns have been reduced to only the Mathematical Methods and Specialist Mathematics examinations from the November 2022 cycle.

We pretty much indicated the exact opposite (p 2):

There is good reason for limiting the review to VCE exams, and indeed in our recent communication with VCAA we have focused upon the exams. Nonetheless, we have also continually emphasised that the exams are the tip of the iceberg, the clearest sign of broader, systemic problems.

Even restricted to the exams (p 2),

We shall provide many examples from the 2022 exams to illustrate these deficiencies, focusing upon Specialist Mathematics and Mathematical Methods, the subjects with which we are most familiar. … It is important to recognise that we have not attempted to be comprehensive, even in considering the 2022 exams, and it would be almost impossible to be so. Every Specialist Mathematics and Mathematical Methods exam that we have seen is marred by at least minor mathematical problems, and most exams contains at least one major error.

It took me and Burkard fourteen pages, and a week, to hammer the exams that we did. Life is not long enough to hammer all of VCAA’s nonsense.

2. Page 1 levels alegations [sic] that “the VCAA has had demonstrable problems with engaging qualified personel” [sic]. Although indicating that a forthcoming argument will emerge in section 7 and 8 of the critique, there is no evidence offered to support the allegation.

If a restaurant serves up a bunch of inedible dishes, you don’t need to look up the chef’s employment history to demonstrate that he is unqualified: you simply have to point to the dishes, demand people taste them. Of course the entire critique is the evidence, which we then summarised (pp 13-14):

First of all, there is a clear lack of mathematical expertise. There are errors on the exams that no competent and attentive mathematician would miss. There are questions and wordings of questions that no mathematician would permit. There is a shallowness to the exams to which any mathematician would object.

Secondly, there appears to have been a process of cultural ossification. There are types of questions and conventions for the wording of questions that we cannot imagine would be acceptable in any other setting. This culture seems so entrenched that even when it leads to questions being meaningless or in error, this can too often go uncorrected.

Thirdly, there appears to have developed within VCAA a self-defensive culture of pedantry and certainty. The principle concern appears not to be to write questions as clearly as possible, not to assess questions with reasonable generosity, but to take an overly formal and legalistic approach, in an attempt to preclude the possibility of complaint. Ironically, this culture, which is highly disrespectful of students, naturally becomes a major source of dissatisfaction.

3. Page 2, paragraph 1 alleges that the VCAA produces “poor exams”. No evidence beyond the authors’ opinions is offered.

The chef thing again.

4. Page 2 paragraph 2 indicates that the authors are “far from being the only critics”. No evidence of the existence of other critics is offered, nor is evidence presented of any criticisms developed by a third party.

Perhaps VCAA had somehow forgotten about John Kermond.

Yeah, we could have mentioned other critics (not just Kermond) and we didn’t. Because we knew they knew.

5. The authors raise concerns that the complete external reviewer report would remain confidential and not open to the authors’ scrutiny, while only an overview would be made available to them.

OK, this is fair and it’s understandable why VCAA felt no need to respond. After all, it’s not like VCAA was planning to simply provide a crappy summary of the review that lies about the review’s contents.

6. Page 2, Section 1 – Introduction, outlines the authors’ stance that the current VCAA Mathematics examinations for Mathematical Methods and Specilaist [sic] Mathematics are “grossly deficient” and “poorly written”. Despite offering their perspectives in support of this stance in later sections, their perspectives are contrary to the advice provided by the writing and development panel, the Examination Study Vetters (ESV) and Study Specilaist [sic] Vetters (SSV), and the psychometric analysis developed from the cohort responses to the questions on the four examinations.

Yes, there are a lot more people on their side, so you have to think that they might be right. Until you look again at all those inedible dishes.

7. Sections 2 through 6 outline the authors’ specific concerns associated with specific questions (see Appendix 2 – Specific responses to alleged errors)

This is the large body. It can wait for another day(s).

8. Section 7 levels the allegation that the VCAA consistently produces examinations that are “seriously, systematically flawed” and produces examination reports which “actively inhibit(s) external feedback”. No supporting evidence is offered.

On the exams, it’s the chef business once again. On the reports, we wrote (p 12),

• It takes over a month for VCAA to publish the exams, whereas it takes NESA a couple
of days;

• It takes at least five months for VCAA exam reports to appear, whereas NESA publishes full sample solutions within about a month, and detailed exam feedback by the beginning of the following school year;

• VCAA reports include only answers and sketches of solutions, whereas NESA publishes full sample solutions.

We cannot know whether this delay and lack of full information is conscious tactic or just self-serving bureaucratic incompetence, but it hardly matters, since the effect is the same: the possibility of and interest in important public scrutiny and debate is diffused. By the time the exam reports have come out, not only has the horse bolted, the horse is well on its way to graduating from university. As for VCE teachers, they are by then too way busy coping with the current year to seek battle on the previous year’s nonsense. And so the cycle continues.

The possibility of public pressure is also complicated by VCAA’s stubborn unwillingness to acknowledge error, much less apologise for it. The willingness of VCAA to obfuscate, or to simply be silent, is legendary. We provide some examples regarding the 2021 exams. …

We then provided four examples from the 2021 exam reports. But, yes, other than the above, we provided no supporting evidence.

9. Section 7 compares the VCAA timelines for the release of external assessor reports with NESA timelines, without acknowleding [sic] the different staffing and governance structures in place for NESA.

Not our problem, mate.

It has to be said, there are strong suggestions that a large source of VCAA’s administrative woes (but not their mathematical woes) is chronic underfunding. But it’s still not our problem, and it does not affect the truth of our claim in point 8.

10. The authors cite examples from the 2021 Specialist Mathematics paper [?] in support of their allegations. The presentation again represents only the authors’ opinion, not the consensus view and was not supported by the psychometric analysis of cohort responses.

This one is the star of the show, and it also forces me to make an embarrassing confession.

One would think that if two evidently strong mathematicians are offering their “opinions” that VCAA’s questions are stuffed then VCAA might pause and weigh up the value of their “consensus”. But no, no pause whatsoever: VCAA’s “consensus” simply won out. It turns out you need seventy mathematicians to win this game. And yes, there is a role for “consensus” in determining mathematical truth, it just doesn’t work the way VCAA imagines.

And now for the embarrassing confession. A few months ago, while trying to figure out who knew what and when, I wrote,

I strongly suspect that at least some VCAA staff also never doubted the errors and their seriousness once the errors were raised, in November 2022.

VCAA’s “consensus” indicates how hilariously wrong I was: to a man, it seems, they were all sold on their crap. That has to be one of the all time wrongest things I’ve ever written. It’s right up there with when I suggested ACARA were being the good guys.

11. Section 8, Conclusion, outlines the authors’ dissatisfaction with being excluded from the examination writing process and alleges that the VCAA examination writers have “a clear lack of mathematical expertise”. No supporting evidence is offered.

On the lack of mathematical expertise, it’s chefs once again. On the claim that we were dissatisfied to be excluded from exam writing, on the contrary, we were hugely relieved. Exam writing and vetting is a stressful, horrible job. What we wrote was (p 14),

During [a meeting with VCAA reps earlier in 2022], we offered to vet VCE exams, for free. Alternatively, we offered to arrange for other mathematicians to vet the exams. Our offer was declined on the spot. The reason, which had something to do with the “democratic” nature of VCAA process, made absolutely no sense to us. Whatever the reason, and whatever the sense or otherwise in that reason, we regarded the outcome, the rejection of our offer, as highly regrettable.

We didn’t want to do it: we were willing to do it, even though we would have hated doing it.

12. The conclusion and critique finishes with a demand that serious changes to the production process for VCE Mathematics examinations occur, under the authors’ guidance, or that if the authors are not utilized in the change, the VCAA acknowledge that these changes are necessary, and develop plans in the public forum to illustrate how those changes will occur.

Of course, we made no such “demand” (p 14):

Serious change is required in the production of VCE mathematics exams, and in the state of VCE mathematics more generally. We offered to assist in that change, and our offer was declined. That is ok, but only if VCAA is also prepared to acknowledge that such change is required, and develops good and proper and public plans to effect this change.

We were not demanding that VCAA cease to suck at writing mathematics exams. We were simply indicating that it is not ok if they continue to do so.

*******************************************************************

What I have written above is about mathematical issues with VCAA, as evidenced by 2022 and some by 2023. Mostly separate to that are the administrative issues, as evidenced by 2023 and 2024. Both were being addressed, although I only have (some) detail of the former. The SR nonsense above is now largely history, at least on those specifics: the Bennett review pretty much confirmed everything that Burkard and I claimed. I have no idea why there was such a huge screw-up with the latter when the former was going so well.

At the moment, VCAA is schizophrenic. Kylie White worked tirelessly to implement the Bennett review recommendations, with dedicated VCAA staff working tirelessly to support her. (I don’t know this first hand, but I know it close second hand.) But VCAA has always had more than its fair share of clowns, some of whom contributed to and wrote SR; there is no reason to presume that these clowns are not still at VCAA. White had to deal with at least a few clowns, and one has to imagine that at least some of these clowns were unhappy with White, not because she was screwing up but exactly because she was doing what needed to be done to reform VCAA. That is always a difficulty with reformation.

There is this media promulgated view that, now White has gone, the reform of VCAA can begin. We’ll see what happens, but I have my doubts, and it can only be harder. White was not the barrier to reform: she was the reformer. Now White is gone, and the clowns are still there. And now the clowns are stronger.