The (Redacted) Deloitte Report

Below is a copy of VCAA’s redacted “independent” review of the “examination development process”, conducted by Deloitte. Yes, Burkard and I started this. No, I haven’t yet had time to look.

42 Replies to “The (Redacted) Deloitte Report”

  1. This is a one line summary of the bit that most readers here will care about:

    “While we have considered the content in the 2022 Maths Exams, we have not included commentary or
    opinion on the mathematical or grammatical accuracy of the 2022 Maths Exam questions themselves, or
    the 2022 Maths Exams Examiner’s Report.”

    Page 7, summary point 2.3.

  2. Biased on my part but I found Appendix 4 interesting and affirming because some of the commentary mirrors the comments I’ve made for my proof-reading work. Since I don’t get feedback on my comments it’s nice to see that other people write similarly!

    1. Ignoring whether the opinions expressed therein should be considered authoritative, does Appendix 4 back up VCAA’s claim that “The independent state bodies found that there were no major mathematical errors within any of the questions”?

      1. RF’s quote from the Report:

        “While we have considered the content in the 2022 Maths Exams, we have not included commentary or opinion on the mathematical or grammatical accuracy of the 2022 Maths Exam questions themselves, or the 2022 Maths Exams Examiner’s Report.” (Page 7, summary point 2.3).

        certainly doesn’t. Nor does Section 4: Improvement Opportunities.

        It looks to me like someone(s) at the VCAA have *ahem* misunderstood much of this report. Or maybe it’s just a case of selective reading. The report casts a pale on a lot of what the VCAA said.

        1. John, I cannot make any sense of your comment. Try to say things straight, without the colour commentary.

          1. I was answering your question by referring to RF’s quote and Section 4 (Improvement Opportunities).

            I think there is very little in the Report, including Appendix 4, that comes even remotely close to backing up the VCAA’s claim(s).

            I do not see how any of the VCAA’s public claims and statements can be reconciled against what the Report says. I think the VCAA either ignored or misrepresented most of the Report.

            Having read the Report, I am not surprised that the VCAA did not want it made public. In some ways, I think it is just as damning as the Bennett Report.

            1. So your “certainly doesn’t”, which is not a full sentence and is not directed anywhere, is in answer to my question? Then what does the RF quote have to do with it?

              “VCAA’s public claims and statements” is too vague. Consider an exact claim of VCAA’s, the one I quoted or whichever you wish. THEN, don’t concern yourself with wishy washy “reconcile”. The questions are, for the statement of VCAA you choose:

              1) Does Deloitte provide any support for this statement?

              2) Does anything in Deloitte undermine or directly contradict the statement?

              To be clear, independent of Deloitte, VCAA’s various “no major errors” statements were false, or worse. And, even from a brief glance at Deloitte, there is *plenty* to criticise, both of the report and in VCAA’s reliance upon it.

              BUT, I want any accusations to be very clearly worded and very clearly targeted. John, you are meticulous when you write VCE notes, and arguably overly meticulous. You, and everyone, should try to be similarly meticulous here, and you cannot be overly meticulous.

    2. Thankyou.
      I find it hilarious that sometimes I’m referred to by name, sometimes I’m redacted, and sometimes I’m assigned a number (Person 3). To borrow from The Prisoner: “I am not a number! I am a free man!”

  3. Quick comments:

    . Recommendations – I’ve never seen anything as weak as “VCAA may consider ….” on many recommendations and none more than “VCAA consider”. This is highly unusual in a report to Govt – while Govt always carefully discusses the framing of recommendations with consultants in a report, it always has the wriggle room to accept or reject individual ones. This probably says a fair bit about the VCAA person in charge of the consultancy – highly unusual.

    . Appendix 1 – I’ve never seen anything like this before and presume it’s just to add a level of appearance of supposed thoroughness. Also a few names have been left in here – not sure if that means anything.

    . Sloppy Appendix 4 – heading not repeated on each page – very simple Word setting. Of itself no importance, but indicative that the report was not properly proof-read (especially when whatever eye-watering fee would have been paid is considered)

  4. I am very happy that Marty’s FOI request for the Deloitte Report was successful. I was also very amazed. I submitted an FOI request for the Deloitte Report on 19 October 2023. My request was refused by the VCAA solicitor on 27 December 2023 (“exempt from disclosure under the FOI Act”). I applied to the Office of the Victorian Information Commissioner (OVIC) on 17 January 2024 for a review of this decision. I’m still awaiting the outcome of my application.

        1. FOI unit (for which read Minister’s office). Or may just be new broom working its way through and hasn’t got to your request.

        1. Technically that may be true but your work has received a lot of publicity in the media and that’s what they may be worried about.

          Or may just be new broom working its way through and hasn’t got to John’s request.

          1. For me, it’s much more than a technicality. I don’t approach the media.

            As for why I got Deloitte and John hasn’t (yet), it might be the difference in the applicant. I would guess, however, that it is more related to the timing of the application: pre-Bennett or post-Bennett.

                1. I disagree. In all likelihood a different person looked at their requests and made different judgments.

                  Bureaucracies (especially semi-functional ones) are by and large not consistent.

                  1. Joe, there’s not a snowflake’s chance that the explanation is “random”. I honestly thought your original suggestion was a joke.

                    1. What’s so inconceivable about it?

                      John’s request gets sent to one person who decides that it’s exempt, and then the solicitor signs off on their decision.

                      Your request gets sent to another person who decides that it’s not exempt, and then you get the report.

  5. So I may not be a popular person but can I ask what has happened here?
    1. There were heaps of mistakes on the Yr 12 maths exams(!) (that bits clear)
    2. VCAA were finally prodded by media coverage and outcry into investigating it
    3. The report done at great expense by Deloitte glosses over the substance of the mistakes and the underlying gaps in knowledge that caused them, and focuses more on waffly ‘processes’ etc etc and exonerates VCAA. ?

  6. Strange that Appendix 4 (Independent Mathematical Assessment By Third Parties) does not include Specialist Maths Exam 1 and the major mathematical error in how Question 3 part(b) was marked and what was written in the Exam Report.

    The question is discussed here:

    PoSWW 32: Confidence Trick

    The potential for an incorrect marking of this question was reported by me to the VCAA several times in November and December, and also in a phone call to a disinterested VCAA Manager of VCE Examinations on Wednesday 2 February 2023. The Exam Report was published a couple of weeks later, the potentiality became an ugly reality and the rest is history.

  7. Hi Joe – you are right that bureaucracies are certainly dysfunctional and they are indeed getting worse than they were. But on important things (read political – that might embarrass them), they are as laser sharp as they can be. All FOI requests (except I think for mundane personal info on yourself), go through the Minister’s office. They are carefully tracked and you can be sure the FOI team has them in mind (worth more than their jobs worth not to).

    I suspect that Marty is right and what happened was that John’s request was dealt with when they were denying there was a problem and anything like that would have embarrassed them and hence the MINISTER. So they blocked it and sent it off to the FOI Commissioner who is so dreadfully underfunded (NO coincidence) that it takes months or whatever to get to it. Hence it was safely out of the way and remains there.

    Marty’s request was granted as the strategy has changed and they have admitted there was a problem. Now it is the Minister’s interest to accept that there was a problem and they are fixing it and this release may help a little in embarrassing the VCAA staff who were responsible for the errors and may be resisting the current changes.

  8. Just a note. Obviously there are issues with the redacting of the report. It is what it is, but no one should be posting on this aspect.

    Stick to the main games, which are:

    a) What are the deficiencies (and strengths, if any) of the Deloitte report?

    b) How does VCAA’s announcements supposedly based upon Deloitte square with the actual report?

    Do not be looking to score cheap points.

  9. I’ve read this twice now and the issue (call it a deficiency if you want, but I think it is more problematic than that) that keeps jumping out at me is that this entire process was to compare what did happen in the exam setting process to what VCAA says should have happened.

    So, if the investigation concludes that VCAA followed its own guidelines, that could be reported as a positive. If the guidelines themselves are a secret from the public, we have to take the report writer’s word for it.

    If VCAA says nothing was/is wrong, that is not the whole truth and not a fair conclusion. If VCAA says that their own policies and procedures were followed, that is a fair(er) conclusion.

    The report makes note that English Additional Language consultants are asked to read the papers and that this is VCAA policy. Fine. If there was a recommendation that qualified Mathematicians read the paper for mathematical accuracy, I missed it. This again is not a weakness of the report as such, because VCAA policies do not require this (as far as I know) but I could be wrong…

    1. Thanks, RF. What you point out is the fundamental absurdity of the Report, an absurdity which was clearly VCAA’s intention all along. I haven’t yet read the report beyond a paragraph here or there, but, despite VCAA’s intentions, I think the report includes important information.

      1. You could well be right – I have had two attempts to read the report now and can never get past the absurdities when I think of what it was like being a teacher during that exam period.

        The report may well include important information. I am yet to find anything helpful though.

        And there is a difference.

        1. I see in the Deloitte Report a bloated and convoluted exam writing process oozing with grandiose self-importance. But there was no room in that quagmire for even one genuine mathematician to provide feedback. It is no surprise that this process consistently led to errors and the coverup/denial of those errors.

          That process couldn’t organise a meat tray at an abattoir.

            1. As a follow-up, in your experience, how does this process compare to the process of writing a university mathematics exam? And which exam, VCAA or university, would you rate superior (mathematical integrity, coverage of syllabus, number of errors)?

                  1. OK, I’ll ponder. It’s a good question and I might post on it. But I don’t think it’s reasonable to compare the school and university exam production processes in the manner you seem to be suggesting, and I don’t automatically buy that the VCAA process is “bloated and convoluted”.

                    Writing and vetting VCE mathematics exams to be sat by thousands of kids, and to keep it all secure is a massive undertaking. I’m going to be very slow to suggest how such a process should be conducted. I have now read Deloitte, and there are aspects of the exam production process, some noted by Deloitte and some not, that seem sub-optimal to me. I imagine the same is true of Bennett (which I have not yet read properly). But I don’t think any such changes, whether recommended by Deloitte or Bennett or (who cares) me, would fundamentally alter the bureaucratic and many-step nature of the process. I don’t see how this can be avoided.

                    I am sure the VCE processes can be improved, but this was always a red herring, a deliberate one thrown in by VCAA. The only major issues are: (a) the deplorable curriculum; (b) the perversion of teaching and assessing by CAS; (c) the lack of solid mathematicians to fix (a), to veto (b) and to write and vet sane exams.

        2. “No Minister, I Beg You,” Replies Sir Humphrey. “A Basic Rule Of Government Is Never Look Into Anything You Don’t Have To, And Never Set Up An Inquiry Unless You Know In Advance What Its Findings Will Be.

          The key in all such reviews is to frame the terms of reference so tightly that you know pretty much what they will say – hence simply check compliance. It’s quite a disaster if such a tightly conducted review (by tame consultants such as Deloitte) found anything at all.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here