Deloitte Dénouement

Long, long ago, when you were little boys and girls, I lodged an FOI. This FOI concerned the Deloitte debacle. My request was broad and complicated but in brief the FOI – in actuality, two related FOIs – requested:

1. All of VCAA’s discussions and analyses, both internal and external by VCAA request, of the five major errors on the 2022 VCE mathematics exams.

2. The Terms of Reference of the review of the mathematics examination development process commissioned by the Victorian Curriculum and Assessment Authority (VCAA) in March 2023, as well as documents prior and subsequent to the commencement of the review, that indicate the formation of and the conduct of that review by VCAA.

That FOI process has now finally run its course. About a month ago, I received VCAA’s re-revised response to my FOIs. As I will explain, there is a key element of the debacle that remains a mystery, and I still had the right of (re)appeal to OVIC. My feeling, however was that VCAA handled my broad and difficult FOIs in exhausting (if sometimes clumsy) good faith, and that nothing was likely to be gained by appealing.

The unexplained mystery is how VCAA came to make blatantly false statements, repeatedly, about NESA’s conclusions on the major errors and, in particular, the role that then VCAA CEO Stephen Gniel played in sanctioning, if not creating, these falsehoods. There is the compounding mystery of why the explanation for the NESA mystery was not revealed in the FOI, but I am convinced that resolving these mysteries requires other methods (or maybe I’m rationalising and I just got sick of the FOI battle). I’m not giving up, and I don’t see myself giving up any time soon. I really dislike being dicked around by arrogant authority.

This post is not primarily about NESA, however, but about other bits and pieces that came to light in the long, long, long document compilations that VCAA provided me. Most of the documents shed little or no new light on the whole absurd episode, and there is a lot or repetition. (For some old light, see here, here, here, here, here, here, here, here, here, here and here.) Nonetheless, there appeared a few new and interesting tidbits.

What follows, accompanied by some brief commentary and explanation, are the tidbits. The dates of the source documents are not always clear and most of the names, of writers and recipients and people discussed, have been blacked out, arguably improperly. Some of the names I know for certain, for many others I have very good guesses, and for others I am entirely clueless. I will not speculate here (and will not permit any speculation in the comments) about any of the people involved.

 

The Plan

In mid November 2022, after (some of) the exam errors came to light, Burkard contacted VCAA (again) to indicate that Burkard and I regarded the situation as unacceptable. VCAA requested that Burkard and I put our concerns in writing, which we did. In mid December, we forwarded our exam critique to VCAA, which was in turn forwarded to Gniel. Gniel wrote an internal email in response (20/12/22):

Thanks, XXXXX

I think we left this with you having a discussion with YYYYY about the proposal from Deloitte. This document is helpful and reinforces that we need to focus on what our process will be to give ourselves the assurance about our work and to be able to respond to the criticism. Although it doesn’t have to be Deloitte we must address the concerns.

Can you please progress this work with YYYYY asap.

Thanks, Steve

This is not awful, and Gniel indicates that VCAA “must address the concerns”, even if VCAA never did and never even gave it an honest shot. Nonetheless, the line about “what our process will be to give ourselves the assurance about our work” rankles, and is a much more accurate indicator of the nonsense that followed. Everything, always, stems from VCAA being way too sure of its own competence.

 

The Cost

In April 2022, Deloitte wrote to VCAA, noting that “co-ordinating the interstate review” was not in the original scope, and that an extra $5,000 on top of the $90K may be required (to coordinate the only part of their review worth a damn).

VCAA first contacted Deloitte about conducting the review in November 2022. Deloitte evidently provided some rough quote, about which someone in VCAA commented internally (01/12/22):

Hello,

Are we wanting to spend 80-100k on this issue (given it has been raised regularly by three people only)?

Up to you, but an obscene amount of money……

Yes, it was an obscene amount of money. Particularly so, given: the review was evidently designed to accomplish nothing; when, contrary to plan, NESA accomplished something, VCAA hid it; and DoE eventually had to initiate a second, proper review.

 

The Experts

In the second half of 2022, VCAA began making their idiotic “no major errors” claims. VCAA also indicated that, apart from QCAA and NESA’s reviews for Deloitte, VCAA had conducted their own review, involving “participants of the exam development process”. So, early on, VCAA asked the clowns who screwed up 2022 whether they still stood by their clownishness. Fair enough, except unfortunately the clowns were still clowns, as was revealed early on by the FOI. This internal review resulted in VCAA’s farcical would-be rebuttal of Burkard’s and my critique.

Did VCAA ask anybody else? It appears that VCAA considered this and may have intended to but it seems that they did not. In March 2023, someone in VCAA wrote (03/03/23),

If we were looking for yet another independent reviewer, I’d recommend ZZZZZ from the School of Mathematics and Statistics at The University of Melbourne. ZZZZZ or someone else in his department would be happy to help. Alternately, I can get someone from the CSIRO to review the questions.

Of course a university mathematics department or CISRO is a better bet than a QCAA or a NESA, but there are a few things to note. First of all, CSIRO is not that good a bet; these guys are smart but they’re more applied sciencey guys, and this matters. Secondly, although Unimelb Mathematics is a strong (if arrogant) mathematics department, it is anything but an idiot-free zone; all departments have their strong guys and their weak guys, and it matters who you ask. Thirdly, whoever the Unimelb person was that VCAA had in mind, the tone of the email does not suggest that this person would have been all that “independent”. In any case, I can find no evidence that VCAA ever engaged an independent reviewer, from Unimelb or CSIRO or anywhere, other than via Deloitte, and there is reason to suspect that they did not.

The email above expresses exasperation that VCAA may need “yet another” reviewer. An email soon after, however, contradicts this (07/03/23):

I had thought we had gotten an independent reviewer to take a look, but this was apparently not the case …

The funniest line about this non-plethora of experts came very early on, when VCAA outlined the review task to Deloitte (27/11/22):

It would be good too to work with mathematicians (maybe from the Mathematics Association of Victoria – MAV) to confirm the quality and validity of our Exams against the Mathematics Study Designs ….. i.e. get opinions from other mathematicians. 

Yep, the MAV would have been a great place to go to find independent experts, unless one was concerned about independence or expertise.

 

The Momentary Relief

In May 2023, VCAA received QCAA’s review of the 2022 exam questions (22/05/23):

Please find a ShareLink file to the response documents from QCAA (10 items).

Sixteen minutes later, the VCAA recipient wrote:

Not sure I want to open them!

Twenty minutes later, a VCAA person gave their preliminary assessment:

Thank you AAAAA,

A very quick scan of the SM2 responses shows alignment with our responses and the commentary does not reveal any issues.

I see this as a small win.

There are three thing to note about this. To begin, yes, it was a small win for VCAA, which was achieved through the utter incompetence of the QCAA reviewers. Secondly, VCAA were evidently very nervous about and keen to see QCAA’s judgment.

The most important thing to note about the QCAA exchange is that there is not a hint of any corresponding NESA exchange, the NESA review being, of course, a massive loss for VCAA. Why not?

It is utterly implausible that VCAA were not just as and even more nervous about the NESA judgment, given that NESA focused solely on the exam questions criticised by me and Burkard. As well, although NESA was too collegial to point out the errors in neon lights (and one of the reasons why QCAA and NESA were such bad choices), it is utterly implausible that VCAA overlooked or misinterpreted NESA’s critical remarks.

This is the sleazy mystery that I am still chasing.

 

The Falsehoods

Beginning in September 2023, VCAA began making false statements about the conclusions of the Deloitte report. With the conclusion of the FOI, two more more iterations of these falsehoods have come to light.

The first such iteration is an internal VCAA document, date unknown, titled “Independent assessment of mathematical content: summary”. This document is a compilation of summary comments on the exam questions from: VCAA’s internal review; me and Burkard; John Kermond; QCAA; NESA; and with a final column for VCAA’s summary conclusion. Screenshots of the summaries of the five major errors are below (and see here, here, here, here and here). The document is utter bullshit, an extended exercise in manipulation and obfuscation, if not outright lying.

The second iteration is a briefing paper between VCAA and the Office of the Minister for Education (27/10/23):

The review of the mathematical content of the 2022 Maths examinations by two equivalent jurisdictions (QCAA and NESA) found that there was room for improvement in the clarity of a number of questions on the VCE Mathematics examination, the reviewers did not concur with the complainants that the examinations contained ‘major mathematical error’ or ‘minor mathematical issues’.

Simply, this was false. VCAA said it many times. Every time, it was false.