Bad Estimates

On 23 November, secretaries of Victoria’s Department of Education appeared before Victoria’s Public Accounts and Estimates Committee, an “Inquiry into the 2021–22 and 2022–23 Financial and Performance Outcomes“. This included the appearance of the Secretary, Jenny Atta, and the Deputy Secretary, Schools and Regional Services, Dr. David Howes. The full transcript is here and answers to the questions on notice are here.

After an introductory presentation by Atta, the questioning was begun by Nationals MP, Danny O’Brien. Below are excerpts from O’Brien’s questioning of Atta and Howes, together with an answer provided later to a question on notice. Emphasis has been added at various points, for obvious reasons.

Danny O’BRIEN: … Moving on, during the reporting period can you give me a bit of background on how the VCE exams were drafted and finalised?

Jenny ATTA: I will ask Dr Howes to supplement my answer, but the Victorian Curriculum and Assessment Authority has a very comprehensive methodology for putting the exams together which involves a range of input, including from experts and expert subject matter specialists. Dr Howes, could you add to that?

David HOWES: Thanks, Mr O’Brien. It is a very detailed and painstaking process. For every study a panel is appointed, and that will include a range of experts in the field and practising teachers to get that balance. It then goes through a vetting process that will include vetting for accessibility for a range of students, and there will be what is called a sitter vetter who will sit the exam as it is presented to ensure it can be completed in the set time and that its level is appropriate for the students. A key consideration is of course making sure that the questions do apply to the relevant study design.

Danny O’BRIEN: Who actually writes the exams ultimately? Those panels?

David HOWES: Those panels, that is right.

Danny O’BRIEN: Are they VCAA employees or departmental employees?

David HOWES: There are a range of people who are appointed – both VCAA staff and a range of external people who are engaged for the purposes of being part of what is called the examination setting panel.

Danny O’BRIEN: Right. It is a joint effort for each exam – it has not got one individual that will write it all.

David HOWES: Correct.

Danny O’BRIEN: What quality assurance processes do you have in place to ensure that mistakes and errors do not occur in VCE exams?

David HOWES: As I broadly indicated, with those other processes that go through, the panel works painstakingly over an extended period of time to ensure that the quality of the questions is right, and then each exam goes through an editing process as well the vetting process. That process that I indicated around making sure the exam can be completed by a well-equipped student in the time that is available is part of the measure.

Danny O’BRIEN: I am more particularly looking at the errors and mistakes not being in the exams themselves. How many sets of eyes would see a final exam?

David HOWES: The short answer is many.

Danny O’BRIEN: It varies probably by subject?

David HOWES: No. The editing process, in my view and my experience, would be thorough and nearly always effective.

[There followed some questioning on confidentiality and exam security, and then O’Brien returned to the exams proper.]

Danny O’BRIEN: … During the reporting period did the VCAA undertake a review of any of the VCE exams for mistakes or errors, and was there an internal or an external review undertaken?

Jenny ATTA: Mr O’Brien, the VCAA, as I am advised, had undertaken a review of some mathematics examinations after a range of issues had been raised with them. I do not have the detail of that with me, but there was a review that they undertook 12, 18 months ago.

Danny O’BRIEN: Sorry, you said you would come back to me with some detail?

Jenny ATTA: I said I do not have the detail on that, but we can come back to you on the scope and timing and nature of that.

Danny O’BRIEN: Is it publicly available?

Jenny ATTA: Is it publicly available? I do not believe so.

David HOWES: My understanding is it was a review of the mathematics exam. They engaged a company to look at that, and one of the questions was ‘Were there errors of mathematics in the exam?’ My understanding – and I am happy to take this one on notice – is that they did not find there were errors of mathematics. There were some recommendations about how some language could be tightened.

Danny O’BRIEN: Any further information you can provide would be welcome. If you can provide a copy of the review, then we would welcome it as well.

[Questioning continued, on other matters.]

********************************************

It is unclear whether O’Brien was subsequently provided with the requested copy of “the review”. The following was subsequently provided as an answer to O’Brien’s question on notice (Question 14), requesting “any further information” on “the review”, which Howes suggested did not find there were errors of mathematics“:

In January 2023, Deloitte Touche Tohmatsu Ltd (Deloitte) was engaged to conduct an independent review of the 2022 Mathematics examination development process to provide feedback and recommendations to improve the effectiveness of the examination development process and the roles of the personnel involved in the examination development. The review is not publicly available.

********************************************

There is plenty one could say about Atta’s and Howes’ responses, especially given the date of the Hearing, but let’s keep it simple:

1) Atta and Howes do not appear to have lied to O’Brien, except in as much as their boilerplate claims of “experts” and “quality” and so forth are demonstrable nonsense, at least in relation to the mathematics exams.

2) It is difficult to believe, and more difficult to justify, Atta and Howes not knowing a hell of a lot more than their responses indicated. Atta and Howes had the responsibility to know that VCAA’s three reviews were farces, and that the 2022 mathematics exams contained blatant, inexcusable mathematical errors.

3) O’Brien would have learned nothing of substance or value from Atta’s and Howes’ responses. Simply, O’Brien asked the wrong questions, and Atta and Howes weren’t going to give him anything for free.

6 Replies to “Bad Estimates”

  1. Hi Marty

    You are right. As an FYI, preparations for these hearings go on for weeks – execs are very well briefed indeed and they would have known that this would be an absolute focus of questions so their answers can be expected possibly even to have been physically rehearsed. But they would definitely have planned this strategy.

    Traditionally the trick was always to tell the strict truth in answer to a question but not more. Though these days lying seems to be much more acceptable throughout our system.

    The department was indeed successful in blocking – the system relies on MPs asking excellent questions. This one didn’t and the opposition is not known for its strength. Good government requires high quality opposition to hold people to account. Possibly if Matthew Bach weren’t leaving, it would have been better handled.

    Yet another reason not to be hopeful I’m afraid ….

    1. Thanks, JJ. Howes’ answers were interesting, because he knew enough for his answers to be thoroughly idiotic. Did he know his answers were thoroughly idiotic? I don’t think he cares.

      Nothing I’ve heard, publicly or otherwise, is encouraging.

      1. Some context: Immediately prior to his current position, David Howes was CEO of the VCAA (July 2017 – Aug 2019). Prior to being CEO he was Executive Director of Curriculum at VCAA.

        On the topic of VCAA CEO’s, after the resignation of John Firth in 2017 (with VCAA for over 30 years), the position of VCAA CEO has been filled by a cast of people:
        Howes, then Claude Sgroi (acting CEO) then Sharyn Donald, then Stephen Gniel (*) as acting CEO when Sharyn went on leave, then Sharyn returned and then left and Stephen started acting again. Now Stephen has moved on to act as CEO at ACARA and Kylie White is the current acting CEO.

        * Who was Deputy Secretary, School Education Programs and Support at the DET at the time.

        1. Thanks, BiB. It would be interesting to know the official, and real, reasons for all the leadership changes.

          I plan to write another post featuring David Howes in the next day or so.

  2. I sincerely hope the Inquiry into Education in Victoria is as hot for some of these people as it has been for supermarket CEOs at the current inquiry into price gouging.

    And yes, it would be very interesting to know if O’Brien was subsequently provided with anything more than further blarney, particularly in light of the Bennett Review. And speaking of blarney …

    If/when the *nudge nudge wink wink* independent review of the 2022 Mathematics examinations conducted by Deloitte is made public, I think the Bennett Review safely ensures it will be classified in libraries as Fiction (subgenre Fantasy). That’s where I’ll be looking.

    1. Thanks, John. One would hope so. And one would hope that Atta and Howes are not permitted to escape with such bullshit answers in the future. At this stage there is no excuse for anyone anywhere pretending that VCAA has not been a basket case.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here