QCAA, VCAA and the Tests of Integrity

We’ve sat on this one for a while, and had not intended to write on it. With recent pronouncements by VCAA, however, we feel we have no choice but to revisit and to update the QCAA story.

Last year there was a blatant error in the grading of the Queensland Mathematical Methods Exam 2. The exam contained a multiple choice question for which the intended answer was incorrect; we posted on the question here. QCAA only learned of the error in April this year, long after the grading and ATARing and course offering was done and dusted.

What was QCAA to do? At that stage, in April, did it matter? The answer is yes, yes and yes. This post is about the third “yes”.

To begin, basic integrity required QCAA to amend the exam report, to indicate that the report had previously been in error and that the exam had been incorrectly graded. QCAA eventually did this, even if their statement was not as clear as it could have been.

Secondly, so that current and future students would not be misled, it was critical for the exam report to be amended to indicate the correct answer. QCAA did this very quickly, thus avoiding the inevitable consequences of failing to do so.

But, thirdly, what about the faulty grading? At that late date was it worth QCAA worrying about fixing the grading of a one-mark question on one exam for one subject? QCAA decided that it was, and they were correct.

After writing about the QCAA error and the amendments to the exam report we were left somewhat puzzled. The amended report indicated that the faulty question had been regraded, but to what end? What did QCAA do with the amended scores for the exam? It wasn’t clear, so, eventually, we asked. We emailed QCAA a while back, noting that we had written about the exam error and asking for clarification on the regrading. A QCAA representative quickly replied, responded politely and made it all clear.

To begin, the rep noted QCAA’s response to fixing the marking guide:

… once QCAA verified the error, the marking guide and subject report were updated as soon as possible to indicate this.

Then it was on to the regrading:

We also amended the affected students’ results and notified them accordingly.

So, the grades were amended and the affected students were notified. But, there’s more.

The Queensland Tertiary Admissions Centre advised that the one mark change had minimal impact on these students’ ATARs, …

Sure, as one would expect. That sentence is not complete, however. It continues,

… with only one student missing out on a higher tertiary entrance preference. 

So, just one student missed out on a course offering. Or, to put it in another way, a student missed out on a course offering. But then,

The student was able to change to their preferred course in the following semester with all credit from their completed studies transferred. 

There you have it. One student’s life was changed, to the course that they had intended and to which they were entitled, and this only happened because QCAA had the integrity to admit their error and to do their best to mitigate the error. This is how an educational bureaucracy should function.

This not how VCAA functions. This is why we felt compelled to write this post. Whereas QCAA manned the battle stations as soon as they were alerted to the error, VCAA has continued to deny the existence of errors on the 2022 VCE exams that are just as blatant, that are at least as bad and arguably worse. Have any Victorian students missed out on course offerings or scholarships as a consequence? Of course we cannot know. But the intrinsic odds of this having occurred appear to be significantly greater in Victoria than in Queensland.

Of course QCAA is not innocent. The exam error was bad and it was obvious. The error should not have occurred and it definitely should have been caught during the grading, at the very latest. But QCAA is clearly aware that such an error is unacceptable. The email ends,

Please also note that QCAA has already implemented a range of corrective measures to reduce the likelihood of this type of error occurring again. This includes extra reviews of exams and marking guides and increasing the size of scrutiny panels.

We’ll see. The number of reviews and the size of the panels matters much less than the expertise and the attentiveness of the reviewers and the panellists. But at least QCAA seems to be taking the matter seriously and sincerely trying. If VCAA is similarly trying we have seen no evidence of it. VCAA’s unwillingness, ever, to admit their screw-ups openly and honestly is strong evidence otherwise.

24 Replies to “QCAA, VCAA and the Tests of Integrity”

  1. Wow! Kudos to QCAA. They made a mistake but wow … they sure should did their best to make things right. Talk about transparency and honesty. That single student has no idea how very fortunate they are to have done their Yr 12 studies in a state where the education authority conducts itself with integrity.

    1. And how refreshingly honest that, after QCAA learned of the error, its response was NOT “Psychometric analysis showed that no student was disadvantaged.”
      (in light of QCAA having given a damn, how terribly wrong such a statement would have been).

  2. Perhaps the QCAA difference is that it is only one error. To admit to four repeatedly would look very bad indeed.

    Suspect that the media is the only solution here. Timed right in the Herald Sun (or Neil Mitchell or Raf Epstein), the Minister could be very annoyed to see it and action might take place. Perhaps especially if there were a storm about Victoria’s need for more maths for the higher economy etc etc.

    1. I don’t think it’s as simple as 1 < 4.

      I know nothing of QCAA’s history, and little of VCAA’s. I have no clear idea when or how VCAA came to be so incompetent and so arrogant, and so plain weird. I’ve witnessed it and written on it and lectured on it for over a decade. However it arose, the culture now seems absolutely entrenched.

      The 2022 errors have been reported on by the media. They were reported on last year, after the exams. Then, recently, based upon a submission to the Upper House inquiry, they were reported on again. Both times VCAA simply denied there were errors, or at least errors of any significance. (Tell it to the Queensland kid.) What, then, is a reporter to do?

      I’ve stayed silent for a patient, painful year, waiting for VCAA to come to terms with their 2022 screw-ups. They didn’t, and I no longer intend to be silent.

        1. Yes, that occurred to me, too, and I think it is an important point. QCAA seems to be aware that they’re learning, and are trying to get it right. (I don’t think they are, but for different reasons.) VCAA seems to believe they have nothing to learn.

          1. Apparently, learning is a 2-way street. VCAA should learn as much from us as we do from them…

            (For those who do not get the reference here, see a comment on a recent post)

      1. Sorry – I didn’t mean it was just that simple and if an article in The Herald Sun can’t move it, then it will be very hard. Keep up the hard work and eventually something may be done.

  3. The other thing that could be done is to find out more about the review by Deloitte – not surprising it’s Deloitte who have no experience in exam development – that’s the way things are done. I can see nothing on the Tenders website – they’re supposed to publish all contracts with a value over $100K there. Not much comes in under $100K for Deloitte of course. First step would be to write to the Minister requesting info. Could escalate to FOI. Takes a fair bit of work all this and risk of alienating bureaucracy. Not for the faint- hearted.

    Of course if there’s another error this year ……

    1. Thanks, JJ. I’ll be writing plenty more on this, and doing plenty more on this. Why do you suggest Minister as a first step and FOI as a second step? Why not simultaneous steps?

      Of course that is a very good point, how this year’s exams go. VCA knows they’re being heavily scrutinised, and one would assume they’ve worked hard to not screw up this year. But, whether they’re capable of that is not so clear.

      1. Hi Marty – writing to the Minister is free. FOI is not. Also may? be slightly more likely to give info in response if FOI not yet in. Never write to department – they track Ministerial correspondence (there are timelines for responding) but not departmental so you’re more likely to get an answer. Let me know if you’d like a hand drafting anything.

        1. Thanks, JJ.

          I might email you offline to discuss, if that’s ok. But a couple quick comments/questions:

          *) FOI is pretty cheap. This one would not be a fishing expedition. (There is some fishing I’m pondering but that’s semi-separate.) So, I’m still not sure why going parallel strategies is not natural.

          *) I don’t know what you mean by “never write to department”. Which department, and why would I be thinking that? If one wants to talk to the Minister, who does one talk to, other than making a request through the Minister’s office?

          1. Hi Marty

            Write to department means addressing your letter/email to the Secretary of the Department or someone in it or VCAA. Same people prepare response, but with priority given to Ministerials. Always write to Minister (or Premier).

            FOI cost should not be huge so go ahead – but don’t be surprised if they charge you hours for blacking out most of the report. Haven’t done a lot of FOI myself but the system is not designed to give out information easily.

            1. Thanks, JJ. Understood. I’ve done a little bit of Federal FOI, and it’s glacial and maddening. You’re fine if I email you, if and when I think it may help?

                1. Thanks. It might have been designed to be glacial and maddening, but I’m not sure it was intended to have such an appalling, industry captured clown in charge.

                  1. Not sure who the ‘appalling, industry captured clown in charge’ is. FOI’s origins are good but the outcomes are mixed. Generally agreed that historians of the future will not be able to research public policy development in future as deliberately little is written down these days. As a public servant, every email you write you think how would this look if it came out?

  4. I think it’s worth noting that by refusing to acknowledge all the mistakes on the 2022 exams, the VCAA has encouraged students (and who knows how many teachers) to be duped and gaslit by these mistakes in the lead-up to this years Specialist Maths exams (starting this Friday 3 November). Who knows what harm this has caused.

      1. Indeed.

        I have been reading about the MAV: https://www.mav.vic.edu.au/About-Us

        In particular, that the “MAV provides a voice, leadership and professional support for mathematics education.”

        I have been contemplating the MAV’s leadership regarding the 2022 Specialist Maths exams. Apart from comments in The Age (10 Nov 2022) by the former CEO Peter Saffin:

        “Mathematical Association of Victoria chief executive Peter Saffin said that teachers and students were right to raise any potential errors in the exam, but that the VCAA had processes to adjust students’ results if their complaints were found correct.
        “In the end they have to be as fair to everybody as they can during the marking,” Saffin said.
        “All students are disadvantaged by an error, so maybe that helps level the playing field.””

        there has been silence as far as I can tell. One can only assume that either the MAV
        1) supports the VCAA’s ongoing claims that there are no errors in these exams, or
        2) is working quietly behind the scenes to hold the VCAA accountable.

        Does anyone have a copy of the MAV solutions to the 2022 Specialist Maths exams? I am curious to know what they say about MCQ 4 and MCQ 19. My understanding is that for the last several years, the MAV solutions are more or less copied from the VCAA solutions. They might shed light on which of the above assumptions is most reasonable.

        1. I would also be curious. As a check on VCAA, the MAV has been absolutely useless for as long as I can remember, but Saffin’s idiotic comments last year were even worse.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here