Aud It Be This Way?

A reader of this blog has alerted us to an audit, conducted in 2010, of VCAA’s administration of the VCE exams. It makes for very interesting reading.

That’s a lie. The audit makes for very boring reading. There is, however, one very interesting line. On the VCE exam vetting processes, the auditor concluded (s 2.2.2),

Vetting processes … have been reviewed and improved over a number of years with a result that no technical errors were identified in printed exam papers in 2009.

That must have been a relief. Although there was this:

Now, to be fair, the issue with the question is not technically a “technical error”. It’s more of a monumental blunder. One might also note that this stuff up was just a one-off in 2009. Well, a two-off. Well, OK, …

In response to this audit and its conclusions regarding VCAA’s exam vetting (and writing) processes, we have two comments and a question.

Firstly, one should never send a mathematical boy to do a mathematical man’s job. (And, certainly, do not send mathematical toddlers.) Undoubtedly, the Auditor-General and their staff would not have had the expertise (nor the time) to confirm for themselves the truth of any such “no errors” claim. If you want mathematics exams properly vetted, audited, whatevered, then get properly competent mathematicians to do it.

Secondly, it seems very likely that either the auditing was incompetent or VCAA actively deceived the auditor, or both. Why? Because it seems impossible to believe that VCAA was unaware, long before the audit, that the Specialist question was stuffed: the error is simply too blatant for someone not to have alerted VCAA, and very quickly. (VCAA has also declared, although much belatedly and unconvincingly, that VCAA was aware that the Methods question was stuffed at the time of grading, and thus long before the audit.)

So, then, did the auditor ever ask VCAA if VCAA had identified “technical errors” on the 2009 exams? If not, then the auditor was incompetent and it also seems highly likely that VCAA withheld highly relevant information. But, if so, then it seems highly likely that either VCAA lied to the auditor, or people within VCAA deceived other people within VCAA.

Thirdly, and finally, the question: is this gonna happen again?

After the 2022 stuff ups, and then the 2023 stuff ups, the Minister for Education has signalled there will be a review of the VCE exams, and VCAA has separately signalled some kind of reviewing. The question is whether any of this reviewing will amount to anything other than the usual whitewashing.

We’ve been open-minded about this, and we’ve been deliberately quiet, allowing things to play out. But the more time goes by without hearing anything of sensible substance from either the Minister or VCAA, or much of anything, the less hopeful we are.

We’re still open-minded. We’re now anything but optimistic.

41 Replies to “Aud It Be This Way?”

  1. As if what happened in the 2009 maths exams doesn’t make reading the review hilarious enough, I think it’s even more hilarious that the review was conducted in 2010 and then in 2011

    the VCAA was forced to apologise “to Melbourne writer Helen Razer after the 2011 VCE English exam falsely attributed an article she had written to ”part-time journalist and blogger Helen Day”

    and then in 2012

    the VCAA was forced apologise and promise that “no students will be disadvantaged after a doctored image depicting a giant robot assisting socialist revolutionaries in 1917 was accidentally used in [the] history exam.”

    Marty, that’s the question:
    Whether any of the promised reviewing and investigating will amount to anything other than the usual whitewashing.

    1. The 2011 and 2012 errors you note are different. They point to a sloppiness of process, rather than a fundamental weakness in the discipline. The sloppiness is not nothing, but it is, relatively, very small beer.

      1. I should have mentioned all the errors on the 2011 Maths Methods exams (particularly Exam 2) rather than focusing on the headlines. Nothing sloppy about those errors, they were good old errors based on fundamental weakness.

        I would have thought all this would prompt a re-audit, but maybe it was too embarrassing.

        1. Yes, the 2011 (and 2010) maths errors are more relevant. But the suggestion that this was “too embarrassing” is hilarious. Simply, no one would have kicked up enough of a stink.

        1. I’m only guessing of course (so don’t blame me!), Craig will need to explain. But maybe you can join your non-existent dots using a cue from the Fin Rev: https://www.afr.com/life-and-luxury/arts-and-culture/only-kraftwerk-can-perform-without-actually-performing-20231207-p5eprd#:~:text=Review%3A%20Kraftwerk%2C%20Sydney's%20Aware%20Super,sounded%20just%20like%20Massive%20Attack.

          “There was no audience acknowledgement until brief bows at the end, and no theatrical attempts to pretend that three of the members were doing much but standing there, pressing buttons, twiddling knobs, and cueing up videos and jumpsuit lights in service of the whole.”

          Personally I think a Milli Vanilli reference is way more appropriate.

  2. No technical errors…

    OK, I have to ask… in what sense is that MCQ 6 not a technical error?

    It is an error brough about by a faulty technique (the setter I am guessing assumed Re(z),Im(z)>0) and therefore the very definition of a technical error.

    Or did I miss the point (again)?

    1. That line (and the whole report) is premised upon the idea that they’re asking about minor, technical issues in the exam processes: slightly better wording, a little more training, stuff like that. I don’t think the Auditor-General thought to ask, “Were any of your 2009 exam questions completely stuffed?”

        1. So they knew the answer they wanted and paid someone to manipulate the data to get that answer?

          Or is this a cynical view?

          Do VCAA genuinely believe that there were no technical errors…?

          1. Too cynical, and misunderstanding of the process. This was an audit undertaken by the Auditor-General. VCAA didn’t choose them or the parameters for the review. BUT, the Auditor-General would have only been looking for minor issues requiring minor reforms.

            There’s no reason I know why the Auditor-General might have or should have suspected major issues necessitating major reform. Plus, the auditing process is structured upon the premise that basically things are ok. If you read the audit, a significant aspect involved self-reporting, backed up by interviews. It presumes a general competence and trustworthiness of the organisation under review.

            But the question is, to what extent did VCAA mislead, or outright lie to, the Auditor-General?

            I think it is unlikely, but not impossible, that VCAA outright lied about the 2009 stuff ups. I also have no reason to believe that VCAA was aware of the systemic problem in the maths exam process: the place has clearly been delusional for a long, long time.

            What I do believe is:

            1) Well before the audit, someone in VCAA knew of the Specialist error. Plus, the amended exam report implies that the VCAA maths guys knew of the Methods error at the time of grading (although I’m not convinced the exam report is telling the truth).

            2) The Auditor-General could not have possibly written that “no technical errors” line if they had been informed of either the Specialist Error or the Methods Error.

            If I am correct then the follow-up question is, where did the break-down occur from (1) to (2)? The only possibilities I see are:

            a) People within VCAA deceived, perhaps by their silence, other people within VCAA on the existence of 2009 exam errors; or,

            b) People within VCAA deceived, perhaps by their silence, the Auditor-General on the existence of 2009 exam errors.

            I don’t know the public service, but I assume this kind of deception is not uncommon. But to a large extent, I don’t care whether it’s cluelessness or deception, since the end conclusion is the same: the audit failed. VCAA cannot be trusted to monitor itself, and the public service is incapable of such monitoring.

            Never send a mathematical boy to do a mathematical man’s job.

              1. I think that’s the main message of the audit, but I don’t think the total message. As I keep saying, it is impossible to believe that someone in VCAA was unaware, basically immediately, that the Specialist question was stuffed.

            1. Re: 1) “the amended exam report implies that the VCAA maths guys knew of the Methods error at the time of grading (although I’m not convinced the exam report is telling the truth).”

              I have compared the original Report to the amended (18 Sep 2020) Report. The following comment was added to the original Report:
              “B was also accepted as it leads to an equivalent expression.”
              Option B is not shaded.

              1. I’m not sure of the point you’re making. VCAA claim that B was accepted but, except for their belated claim, the circumstantial evidence suggests otherwise. What are you adding to this?

                1. We know reports get amended because we see a “(amended date)” next to the report file name. However, unless you have the original report and are willing to painstakingly compare the original and amended reports line by line, it’s impossible to know the amendment(s) made.

                  For this particular report, the adding of the comment as an amendment is not obvious. That is, there is nothing to suggest that the comment was not part of the original report and that some other (maybe minor) amendment was made.

                  So my point is that the comment was added many years later, it is an extremely “belated claim”, and the truth of the comment must be judged accordingly. Together with the fact that, for some reason, Option B was not shaded when the comment was added.

                  On a related note – I saw it suggested somewhere that the Wayback Machine could be used to allow comparison between original reports and amended reports. It would be a potentially very tedious task to do this, but it would also be very interesting to know what changes are made. Of course, it would be much easier to know what changes were made if they were somehow highlighted in the amended reports.

  3. “But the more time goes by without hearing anything of sensible substance from either the Minister or VCAA, or much of anything, the less hopeful we are.”

    A while ago we contacted the education minister and the VCAA about our suggestions and unsurprisingly we have got no reply and none of the requests have been met. The shadow education minister did get back to us and said she’d keep us updated when she hears a response from the government to her speech in parliament, and there has been no update.

    We were interested to hear last week the VCAA admitted to another error, in the chemistry exam. This one was pretty blatant and got in the media so we were shocked it wasn’t addressed that night or the day after like with those couple errors in the maths exams that they acknowledged, but the fact that they eventually did so suggests they are under pressure.

    1. Thanks, HSA. The current silence of the Minister and VCAA may not be a bad sign, but it’s definitely not a good sign. Do you have a date or a link to Wilson’s speech?

      1. There is no reason for the Minister to telegraph his intentions, especially to the people who raised the issue. Silence is logical – this is likely to be fixed behind the scenes, with the least embarrassment possible (especially since Labor has been in power for ages). First he needs to work out exactly what he wants to do and then take action would be my guess. At some point it may be that an action plan to ‘improve’ exam processes is quietly implemented and also a new head of VCAA appointed.

        If the errors are not fixed, the ambitious Minister will be said to have failed – remember that there is an annual very concrete and public check. Furthermore that check occurs just before the end of November when elections are held. And fixing it won’t hurt his campaign for the top job. Very strong reasons to fix it.

        1. Thanks, JJ.

          1) I don’t expect colour commentary as this is done, but I expect something more than “we will have a review”.

          2) I think the Minister has more serious intentions than VCAA last year, if only for the cynical-political reasons that you suggest. That doesn’t mean he, or whoever he is handballing the review to, has any clue whatsoever how such a review should be conducted. Is there any reason, for example, to trust Atta or Howes to be shepherding such a review?

          1. Hi Marty

            My guesses ….

            1) Never commission a review unless you know the outcome (would be surprised if there’s much more publicly at least until they know what they are doing – in whose interest would it be?).

            2) He is a new Minister and so won’t know the department. Howes is a former CEO of VCAA so probably not best placed to do the review (but who knows). Jenny Atta looks like a Treasury person so will rely on content experts and is anyway too busy to oversee this but she will need to be onside.

            The Minister said it was the VCAA Board Chair (who appears to have content knowledge) and a department person. If I were doing it, I’d have an Exec overseeing the review reporting to a Dep Sec and have the Exec supported by a team and of course a Steering Committee. I’d keep it totally internal to DET with communications managed very very carefully.

            1. Hi, JJ. I’ll reply in detial tomorrow, but one line I didn’t understand:

              “The Minister said it was the VCAA Board Chair (who appears to have content knowledge) and a department person.”

              What does “it” refer to? *What* is the VCAA Board Chair?

                  1. Sorry Marty – got my wires crossed – looks like not Minister speaking …

                    13 Nov
                    “https://www.theguardian.com/australia-news/2023/nov/13/vce-wrong-exam-victorian-government-chinese-language”
                    The body confirmed to Guardian Australia that its board chair would work with the department of education to conduct a comprehensive review of the vetting and proofing process for exams in light of the errors.

                    17 Nov (paywalled but this is still there)
                    “A plan for an external audit into the laundry list of VCE exam errors has been shelved”

                    https://www.heraldsun.com.au/subscribe/news/1/?sourceCode=HSWEB_WRE170_a_GGL&dest=https%3A%2F%2Fwww.heraldsun.com.au%2Fvictoria-education%2Fyear-12s-start-petition-against-vcaa-amid-another-stuff-up-on-final-day-of-exams%2Fnews-story%2F64ed771b3470194a3350d5e256faf2bc&memtype=anonymous&mode=premium&v21=HIGH-Segment-2-SCORE

  4. I know Marty likes his blog titles. But maybe he wonders whether people get the pun.

    I know he was particularly pleased with https://mathematicalcrap.com/2022/11/09/witch-87-cannery-row/

    which was onion-like.

    Anyway, I digress. Like many things, some things for some people are more obvious than others. So, for what it’s worth and to offer Marty some contentment:

    1) “Aud” is an abbreviation of “Audit” and is semi-homophonic with “Ought”, and

    2) “Aud it” combines as “Audit”.

    A clever play on words. And the answer by the way is no. We should all make sure that it aud not be like this. The day is still young.

  5. In reply to JJ (and thanks again).

    I do not believe the VCAA Board Chair will have any meaningful role in anything, and she should not.

    I think the Herald Sun article is a beat-up, or at least is pretty clueless. I think there are two separate processes going on: internal VCAA reviewing; the preparation of an external independent review, put together by VCAA. The Herald Sun article seems to confuse the two.

    It is unclear what either review will amount to, and how many confidence, if any, one should have in either review. At the moment, I’d go with “not much” and “not much”.

    1. Regardless of opinions on Herald-Sun articles, at least the HS is maintaining attention on the issues. I think that’s a good thing for many reasons and outweighs any perceived weaknesses in the articles.

      I think part of the reason VCAA has dodged the bullet for so long is the lack of media attention. Imagine if there had been such sustained attention 15 years ago. In my view 2023 was no worse than 2022 (and it was possibly better – the many maths errors in 2022 were much more egregious). Or 2021 etc.

      1. I agree. The Herald Sun has had some overreach, and I think the article JJ linked to is confused, but the Herald Sun has mostly been strong and good.

        There is no question that it is only because of the media attention that VCAA, DoE, Minister are making any moves towards reform. The Guardian and the Age and ABC radio have also been good at times, but HS kicked it off.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here