Deloitte, Gniel and VCAA’s Falsehoods

The background to this post is here.

Although the Deloitte review of VCAA’s mathematics exams was a sham, designed to conclude nothing of substance, the final Deloitte report nonetheless contains some well-hidden but damning conclusions. Having finally been permitted to read the report, it now seems clear to me that VCAA repeatedly lied about these conclusions: to me and Burkard; to John Kermond; to at least two media outlets; and thus, by extension, to the general public. It also seems clear that the fundamental responsibility for what was, at the bare minimum, a shameful deception sits with Stephen Gniel, then CEO of VCAA and now CEO of ACARA.

VCAA’S Statements on Deloitte

VCAA received the Deloitte report in August 2023, although VCAA neglected to tell anyone, much less release the report. Our first news of the report was in September, when Herald Sun journalists followed up on John Kermond’s submission to a legislative assembly inquiry. Kermond’s submission included claims of errors on the 2022 Specialist Mathematics exams, together with a cataloging of Kermond’s failed attempts to have VCAA concede that these were errors. These claimed errors were a subset of the “five major errors” that formed the cornerstone of Burkard’s and my critique.

In early September and in response to the Herald Sun‘s queries, VCAA prepared a media release. The media release noted the “independent review conducted by Deloitte” and, on the question of 2022 exam errors, stated,

The VCAA has confirmed that the [2022 exam] questions did not present errors. … The questions identified [by Kermond] as erroneous were reviewed by maths specialists from two other jurisdictions and … the [Deloitte] review found that the questions did not have serious errors that impacted students. [emphases added]

The Herald Sun‘s subsequent report, on 7 September, quoted the media release on the errors, essentially verbatim.

On 8 September, local ABC radio followed up the story, including a news bulletin:

… the Victorian Curriculum Assessment Authority … [has] confirmed there were no errors in the 2022 exam [sic]. [emphasis added]

It is unclear whether ABC’s statement was based on the same media release or on some further communication from VCAA.

In October, VCAA sent Burkard and I, and separately John Kermond, a short summary of the final Deloitte report. The summary noted that “two independent state Education bodies” had reviewed the 2022 exam questions, with hindsight an obvious reference to QCAA and NESA (see the background post). The summary then stated,

The independent state bodies found that there were no major mathematical errors within any of the [2022 exam] questions, … [emphasis added]

At the time, Burkard and I knew, of course, that these statements by and due to VCAA were Alice in Wonderland madness. There was zero doubt that the claimed 2022 errors were indeed errors, and major ones, as later testified to by 70+ academic mathematicians signing our open letter, and by the Bennett report (p 6). (I strongly suspect that at least some VCAA staff also never doubted the errors and their seriousness once the errors were raised, in November 2022.)

Lacking access to the Deloitte report, however, what we could not then know was whether VCAA’s statements were nonetheless true. That is, we had to consider the possibility that VCAA had accurately summarised the madness, that the “independent state bodies”, quite plausibly as ignorant as VCAA, had concluded the exams “did not present errors” or whatever. We had no means then of proving otherwise. Now we do.

NESA’s Conclusions on the 2022 Exam Errors

The body of the Deloitte report is useless on the question of 2022 exam errors (and in every other manner). The Deloitte authors indicate that they “considered” the 2022 Maths Exams (p 7), but there is not a single statement of fact, much less any analysis or a consequent recommendation. The authors simply point to a “summary” of QESA and NESA’s “feedback”, contained in Appendix 4 of the Report (pp 44-53). Which Deloitte garbled.

Appendix 4 contains feedback solely on Specialist Mathematics Exam 2, with the other exams omitted entirely, presumably in error. (Plus, p 45 is repeated.) Still, the appendix provides us with the feedback on three of the five errors, which is enough.

QCAA’s feedback is useless, the inevitable result of a lazy once-over of the entire exam and an archetypal example of education’s culture of mediocrity. NESA, however, is damning: on all three errors NESA agreed entirely with me and Burkard, and with reality:

1. On MCQ4, the multiple choice question with no correct answer, NESA wrote (p 44),

Since it is not stated that Im(a)≠0 and Im(c)≠0 none of the statements are necessarily true. [emphasis added]

2. On MCQ19, the meaningless confidence interval question, NESA wrote (pp 46 (skip 45 repeat) 47),

Confidence intervals are a range of values that the true value is likely to lie in. This question states the true value of the population mean for the mass so a confidence interval is not necessary as the exact value of the average cost can be calculated by substituting m = 7 into the equation for C. [emphasis added]

To calculate a confidence interval for the population mean, we would need a value for the sample mean, which was not given. If it was intended that 7 is the value of the sample mean mass (not the population mean mass) for the sample size of 100 then… (see solution) [emphasis added]

3. On Q6f, the unanswerable probability independence question, NESA wrote (p 52),

“It is not clear from the layout of the exam that the blurb above part e) is also related to part f). There should also have been mention of the independence of the variables for this question to be valid. [emphasis added]

Hilariously, on this final question, NESA then adds,

This is discussed in detail in this blog post: https://mathematicalcrap.com/2022/11/09/witch- 87-cannery-row/ [emphasis added]

Yes, NESA referred to our blog post, in which the question was ripped to shreds by commenters. Words cannot describe the surreality of this. Hunter S. Thompson is all that fits:

 

Did VCAA Lie About Deloitte?

It seems difficult to conclude otherwise. VCAA’s public statements on the existence of 2022 exam errors, “major” or otherwise, are clear falsehoods, not coming within a thousand miles of a faithful encapsulation of NESA’s conclusions. Further, I cannot see how these can be other than conscious falsehoods: that is, lies. I cannot conceive of how VCAA would not have been spider-sense alert to NESA’s or QCAA’s conclusions on the five major errors. Even if they were asleep, one has to imagine that NESA pointing to this blog would have woken up VCAA like a bucket of water to the face.

Who is Responsible?

I do not excuse anybody involved in this farce, but it seems clear to me that the large majority of responsibility lies with Stephen Gniel. Of course, as CEO, Gniel bore solid responsibility for the actions of his senior underlings, but one can also reasonably suspect more active responsibility.

Burkard and I had already been in contact with Gniel in May 2022, with our concerns about Deloitte and the review process; Gniel replied in June, clearly informed about (or at least signing off on) the review process details, and explicitly acknowledging our central concern of the exam errors. Then, for the production of VCAA’s September 2023 media release, “the office of the CEO” was definitely kept in the loop and was seemingly actively involved. Moreover, on such a public and heated matter, it is difficult to imagine that any public statement, including VCAA’s summary of Deloitte to me and Burkard and to Kermond, would not have been, at the bare minimum, signed off by Gniel. I cannot prove Gniel’s active involvement in VCAA’s falsehoods. I can see no good reason to doubt it.

If the facts of this farce are as they appear to me to be, I cannot see how Stephen Gniel’s position as CEO of ACARA is tenable.

4 Replies to “Deloitte, Gniel and VCAA’s Falsehoods”

  1. 1) Your suspicion that “at least some VCAA staff also never doubted the errors and their seriousness once the errors were raised, in November 2022.)” is interesting. I have sometimes wondered why the VCAA Mathematics Manager was totally silent throughout the whole sordid fiasco. I wonder if he was deliberately muzzled by the executives above him.

    2) “If the facts of this farce are as they appear to me to be, I cannot see how Stephen Gniel’s position as CEO of ACARA is tenable.”

    The same observation might be made about the

    “selection panel comprising ACARA Board members and a representative from the Australian Government Department of Education” that “selected its preferred candidate.”
    And then, of course, the Cabinet at federal government level rubber stamped him.

    3) I find the (undisclosed) amount of tax payer money that was spent trying to maintain these lies extremely troubling.

    1. 1) This is too conspiratorial (but not by much), and too emotive. (And watch your step.)

      Once the errors became contentious, and definitely once it became public and hostile, no one was going to speak for VCAA on the issue except the higher, authorised people. This would be standard practice for any body and it is absurd to describe the people as being “muzzled”.

      The question for me is not the lack of public comment, but what was thought and/or said within VCAA. I do not believe all VCAA staff bought the “no errors” line. Ever. So, assuming this, the question is whether such staff kept their opinions to themselves or whether they were ignored/overruled. I have no idea which.

      2) A lazy, ridiculous comment.

      Deal with individual responsibility (when you can, for people sufficiently high, on your own blog), not collective guilt. As a group, the ACARA Board is a useless clown circus but, other than the Chair, NSG Scott, we have no idea who is to blame for the circusness.

      3) Yeah, the cost is a pretty obvious fish in the barrel, although I’m honestly not that concerned.

      I’m much more concerned with the the fact that Deloitte was a farce which delayed the truth coming out for over a year. It is almost certain that by June 2023 VCAA had confirmation that the 2022 exams had been misgraded, still way too late, but not clearly too too late for the 2022 VCE students who had been cheated.

    2. Hi John – the minute the issue became political – ie you, Marty and Burkard raised anything, the issue would have been kept away from the Mathematics Manager, whose job is to be more factual than political.

      The notion of lies is also interesting. In the old days, the public service code about written materials was to write the portion of the truth as fitted the story (ie made us look good). You didn’t tell lies (except by omission). It was a bit of a legalistic type approach I suppose. Media releases were always a bit of a battle with the internal journos to make sure that the signed off wording fitted this code as the journos were looser with their words. You would need to check the written record carefully to see if this code was fulfilled. In this checking, you need to check EXACTLY what is said, not what a reasonable person might infer.

      Of course that may have changed.

      1. Thanks, JJ.

        On the reluctance to avoid falsehoods, conscious or otherwise, of course I know that game, and I think it is alive and well. That is what was so astonishing about Appendix 4. After I posted Deloitte, one guy wrote to me off-blog and said his jaw hit the floor.

        I still find it difficult to believe VCAA blatantly lied. But I simply don’t know what else to think.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here