The New Report on the VCE Exams is Out

The report of the VCE exam review conducted by Dr. John Bennett has been handed down. The media release is below and the Executive Summary of the report is here (and in Word). Presumably, there is no intention to release the full report. (23/03/24. And my presumption was wrong. See the update below.) I haven’t yet had a chance to look at anything, and I may update this post later.

UPDATE (20/02/24)

VCAA’s acting CEO, Kylie White, gave a lengthy ABC interview this afternoon. There are also reports in the AgeHerald Sun and EducationHQ.

I’ll leave off writing any detailed thoughts about the report until the dust and my brain have somewhat settled. In brief, while there are significant nits to pick the report is very good on some important recommendations. In particular: employ competent mathematicians; and report in a timely manner. These are not magic wands, but they would be big and important steps.

UPDATE (23/03/24)

The full Report (modulo “a small number of redactions”) of the VCAA Review Panel is now available, announced here, and PDF here. As BiB noted, it is impressive that the Minister kept his word and did this. Not exactly what we’re used to.

I’ve had no time to look at the report, but will try to do so soon.

 

VCE Exam-Setting Process Set For Improvement

20 March 2024

The process for setting Victoria’s 2024 maths and chemistry VCE examinations will change, following an independent expert panel handing down its report into errors in a small number of last year’s exams.

Minister for Education Ben Carroll welcomed the report from the panel, setting a clear expectation that the Victorian Curriculum and Assessment Authority (VCAA) moves quickly to implement all the recommendations to give VCE students, teachers and assessors certainty in this year’s examinations.

A three-member panel was established following errors in the General Mathematics, Mathematical Methods, Specialist Mathematics and Chemistry examinations last year, and a small distribution error with the Chinese Second Language exam.

Chaired by Dr John Bennett, the panel found the VCAA already has comprehensive policies and procedures for developing, reviewing and distributing external examinations – and made six recommendations for improvement.

The recommendations include increasing the representation of qualified academics on the Mathematics Examination Development Panels, steps to strengthening the examination-setting process for the Mathematics examination papers, as well as the review process for Mathematics and Chemistry exams.

Other recommendations include updating training and guidance for the VCAA’s editors and desktop publishers, updating guidance for staff at examination centres and publishing examination reports in a shorter timeframe.

All students who sat the compromised exams in 2023 were awarded a correct score for the questions, to ensure no student was disadvantaged and the assessment process remained fair, valid and reliable.

To see the Executive Summary and recommendations of the report, visit vic.gov.au/independent-review-vcaa-vce-examination.

Quotes attributable to Minister for Education Ben Carroll

“We support all the recommendations from this independent review, and will work to implement as many as possible before the 2024 examination period – so students and schools can have faith in the integrity of the VCE.”

“We thank Dr Bennett and the panel for their hard work and recommendations as we work to reduce the risk of any errors on VCE exams.”

Quotes attributable to Panel Chair Dr John Bennett

“I thank the VCAA and all those the panel spoke with for their full cooperation and constructive input to our report.

“Overall, I was impressed with the processes that the VCAA have in place for the setting of the VCE examinations. I trust the panel’s recommendations will assist the VCAA to strengthen its policies and procedures even further.”

29 Replies to “The New Report on the VCE Exams is Out”

  1. “The recommendations include … steps to strengthening the examination-setting process for the Mathematics examination papers,”
    Not a good start

      1. It’s not standard English to my knowledge.
        Should be “steps to strengthen” or “steps for strengthening”. Maybe it’s an Americanism?

        1. OK, sure, that’s what I figured. It’s at best clumsy, and probably ungrammatical. At the moment I’m more concerned for substance. If VCAA were to write “We seen there was errors on the 2022 exams”, I’ll take it.

            1. Hi Tom – from experience, it’s so hard to see all the typos in a report, particularly when it’s high profile and rushed. You read it again and again and still miss some.

  2. A few things from the summary you’re probably interested in:

    “Balancing these 3 factors [study design specs, technical correctness of phrasing and layout, expected mathematical proficiency of students] is challenging, and the Panel found that in the case of some
    questions in the 2022 and 2023 VCE Mathematics examinations, mathematical precision and
    ‘correctness’ have been given a lower priority in the interests of accessibility for test takers.” p6

    “Regarding the 2023 examinations, concerns had been expressed by Mathematics academics
    and others around 19 questions. The independent Mathematics experts engaged by the Panel
    found that:
    • Of the 5 questions of concern in the General Mathematics papers, all 5 were found to
    be poorly worded and could have been improved. Two questions had more than one
    answer and another had an error in the question, which was acknowledged and dealt
    with by the VCAA.
    • Of the 14 questions of concern from the Mathematical Methods and Specialist
    Mathematics examination papers, 3 were found to be acceptable by all 7 independent
    experts. However, of the remaining 11 questions, at least 5 of the experts (and in
    some instances 6 or 7) found the questions to be poorly worded or to use poor
    terminology; not to be aligned with the Study Design (regarding a term used in 2 of the
    questions); or, in the case of one question, to have a wrongly labelled graph, which
    was acknowledged and dealt with by the VCAA.” p6

    “In most cases, the errors identified in the Mathematics papers were present from the earliest
    drafts. These were not detected in any of the multiple opportunities for review and revision
    that were undertaken.” p6

    Recommendation 6a
    “i. Publish the VCE Mathematics examination paper on the VCAA website within 5
    days of a paper being sat.
    ii. Publish the final version of the Marking Guide used to assess students’
    performance on the VCAA website within 5 days of the end of the marking of
    each Mathematics examination.
    iii. Publish the report prepared by the Chief Assessor on the VCAA website before
    the end of January the following year.” p10

    1. Thanks Alex. Recommendation 6a is good. Releasing the marking guides would be a big step forward for equity and clarity. And having everything released in a timely manner would be great – the examiners’ reports are still not released from last year, I had a student expressing frustration at that yesterday.

      1. In fairness to VCAA, I assume this year’s exam reports have been delayed while VCAA has been recovering from the bombardment, and dealing the review panel et al.

  3. Page 5:

    “These 4 errors were dealt with in
    the administration of the examination papers and/or in their scoring. The Panel found that
    the actions taken by the VCAA in dealing with the impacts of these issues on students appear
    to be appropriate under the circumstances and are similar to what other comparable
    Australian jurisdictions would do.”

    Does the Queensland exam error and subsequent action not contradict this?

    Or does Queensland not count as “comparable”?

    1. The handling of the Queensland error should be compared to, if anything, the 2022 VCE errors (when VCAA went full ostrich).

      In 2023, the VCE errors were, in the main, acknowledged and dealt with promptly and as reasonably as could be, and before the grading was finalised. I think there were at least a couple 2023 errors that were denied to be errors by VCAA, but in principle they handled the post-exam stuff properly.

  4. I heard about this stuff on the radio as I was driving around an hour ago, and I said out loud (to myself!), “Good on ya Marty”, because I think a lot of this progress has come from your tireless drum banging. Good on ya Marty. Bang away.

    And further, conversationally, I’ve been in classrooms over the last few days watching students do various types of Naplan tests. It really is very strange to watch students stare at their screens and tap away, and to have pretty much no idea as to what they are actually doing. I tried to read a couple of questions over students’ shoulders, and was only able to get as far as “Martha wants to buy ribbons to decorate…” To which I mumble under my breath, “No she doesn’t”.

      1. I second ThatGuy’s assertion that you can take a lot of credit here, Marty, for being noisy and irritating, and *persistent* enough that the right people felt that something had to be done.

        1. Thank you. Do not discount Burkard’s work and also, especially, John Kermond’s. Kermond was the teacher who put his neck on the line.

  5. “All students who sat the compromised exams in 2023 were awarded a correct score for the questions, to ensure no student was disadvantaged and the assessment process remained fair, valid and reliable.”
    Get stuffed.

    Although it is nice to see an acknowledgement that 5 questions from 2022 were “fundamentally flawed”. Strikingly similar to “unacceptably flawed”.

    1. Yes. I wish the authorities would stop it with this blatant lie.

      I think VCAA did what they could (modulo questions that were more borderline in error). But if there is an error, it is then simply impossible to make the grading “fair” or “valid”.

  6. Following a brief skim, my guess is that this is as much as was possible under the circumstances. Pretty damning indeed of VCAA! Well done Marty and Burkard and John (and others no doubt).

    1. Thanks, BiB. Will add a link in the post and look tomorrow. Had ero time so far.

      I agree on the Minister keeping his word. The comparison to the bullshit summary of the Deloitte thing is pretty stark.

    2. Thanks again, BiB. I’ve updated the post with the link. Still had zero time to even glance at the thing.

        1. Any chance of what? Looking at Bennett? No, not yet. (Please don’t use “Anonymous” as your name for commenting.)

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here