Presumably many readers of this blog will have read or heard of Susie O’Brien‘s new Herald-Sun story, on the “compromised” VCE exams, and if not they will probably hear plenty on it soon. It would appear that exam cover sheets VCAA released had hidden exam questions that could then be unhidden. VCAA then scrambled to adjust/replace the accidentally revealed questions before the exams were sat. This is undoubtedly a screw-up. It is a legitimate story. It is also, at least in the case of Specialist Mathematics, a beat up.
The first we knew of this was a couple days ago when, on the Secret Exam 1 post, commenter Allan noted that VCAA had accidentally released an exam question, and then “didn’t do much to alter the style” for the replacement question. Which is a bit of an understatement. Here is the eventual exam question:
The original, leaked question was identical in wording, except with the function originally 
and the factor originally 
.
This is obviously outrageous, so much so that Allan’s comment was flooded with replies. Well, to be precise, it was flooded with one reply. Which was me, going “Whoops”.
And, definitely, it was a whoops, and a big one. But if readers thought it was a big deal, they didn’t bother to say so on the blog. I do not believe it is a big deal. Others do.
In her report, O’Brien quotes John Kermond:
“The rule for the function in the actual exam was different, but similar enough so that both parts offered the same solution pathway. A student having prior knowledge that a question of this type was on the exam would obviously be advantaged over a student who did not know.”
It may be obvious to Kermond, but it’s not obvious to me that the advantage is any more than trivial. Yes, the exam question and the leaked question have “the same solution pathway[s]”. But it is also “the same solution pathway[s]” for plenty of exercises that any Specialist student would have practised over the year. None but the very weakest student would see such a question and not recognise quickly, probably immediately, at least one way to proceed. The exam question is easy and very standard, and that is central to evaluating VCAA’s makeshift fix of the exam.
This obviously should not have happened. In terms of the fairness of the Specialist exam, I think it is no big deal.
UPDATE (14/11/24)
The Age also now has a report on the story.
UPDATE (14/11/24)
And, The Guardian.
UPDATE (14/11/24)
And, The Australian.
> None but the very weakest student would see such a question and not recognise quickly, probably immediately, at least one way to proceed
If the question wasn’t testing anything, it shouldn’t have been included in the exam. The simple fact is that it was included because VCAA believed it was testing something. Maybe most people would immediately know how to proceed, but some definitely wouldn’t, especially under time pressure
I think it’s obvious that not all students got that question correct, and I think it’s obvious that the percentage of students who got it correct will be significantly higher for those who had seen the exam in advance. And, regardless, VCAA should have fully changed the question to maintain the exam’s integrity.
Yes, the article is overblown, in the case of spec. For various other subjects it is definitely not overblown, and hopefully Susie’s “beat up” will result in VCAA changing next week’s exams more than one word
You agree with Kermond. Fine. I do not.
It seems to me like VCAA‘s response was unfortuneately to trivialise the question to the point where evey competent student would recognise the factor and get the marks, whether they‘ve seen the leak or not. The 3z+2i factor sticks out like a sore thumb whereas the original equation better „hid“ the factor so the student to actually needed to be aware that it may be possible to factorise this way.
I don‘t think such a shortcut could be considered obvious to all students. Definitely somewhat routine, but something easily missed or forgotten during the year. There are of course the more normal, routine methods of finding a factor, but these take up precious time.
It‘s a shame that they just trivialised the question rather than replacing it
It’s a screw-up, no question. It would appear that VCAA became aware of the screw-up in October, and so presumably had very little time to fix things. Of course it would have been preferable to have replaced the question entirely, and in principle it would have been easy to do. (A teacher could do it in half an hour: five minutes to come up with a new complex factoring question and twenty-five minutes for reprinting.) In practice, I assume the cumbersomeness of the exam production and distribution process precluded that.
Ah no, my error. Thanks, RF. Will fix.
Did you mean
where you wrote 
or did VCAA make an error in the “draft”? (sentence immediately after your red box).
Also – it is a pretty common exam question, so… could it have given someone an advantage? Maybe. Significantly? I doubt it.
Too many “what if” questions here for me to think it will have any bearing on anything other than perhaps VCAA using thicker paper for their exam covers.
Thanks, RF. My goof, now fixed (and your previous comment deleted).
Fixed. Thanks again. (Do you want me to delete one of your comments as a repeat?)
Yes please. Internet dropped out…
This may not relate to this thread but I just turned up at school when some teachers just inform me that VCE Methods and Business management were leaked. I did not know how reliable it was but my heart sinks. Does anyone hear anything?
Thanks, Victoria. I assume it is the same thing, but I don’t know. Again, if it is the same thing, then I would assume that the original questions have been leaked, and have now been replaced.
Business management leaked, I’m not sure whether it was changed. The Math Methods cover sheet wasn’t saved to the Wayback Machine, so it’s unlikely that it leaked widely (it’s definitely plausible that a few students were aware of the questions on the first page of each section)
“exam cover sheets VCAA released had hidden exam questions that could then be unhidden”
Please explain? Are we talking about a digital file that was somehow engineered to only show the first page?
Click on Allan’s comment, go to his link and follow his instructions.
VCAA publicly releases the cover and instruction sections of all of their exams, including the first page with questions redacted for students to look at the questions beforehand. Some of these files, when opened in libreoffice (or probably word, haven‘t tried) had the questions first page of the exam or a draft version of them
It was accessible in any PDF viewer simply by highlighting text and copy-pasting it
VCAA maintains a page (https://www.vcaa.vic.edu.au/assessment/vce-assessment/past-examinations/Pages/ExaminationMaterialsIndex.aspx) which has the cover sheet and first page of every exam, so that there’s no surprises about the rules. If you look at https://www.vcaa.vic.edu.au/sites/default/files/Documents/exams/exam-covers/mm1-cov.pdf, for example, you’ll notice that the 2nd page is mostly blank, which is where they’ve deleted the actual exam questions that are on the first page of the exam. The issue is that it was previously possible to unhide those questions by using “select all” and copy-paste. It should be noted that only the first page of each exam was leaked (in exams with multiple sections, the first page of each section was leaked).
I’ve been studying cyber security lately, and it’s a well known fact that redacted text is often revealed through copy-paste. Most file editors (including Adobe Acrobat and Microsoft Word) have a “sanitise” feature which scans files for hidden information. That should ALWAYS* be run after removing sensitive information, and this wouldn’t have happened if they had run it (I’ve tested that). The fact it wasn’t run is an indication that VCAA don’t have a proper cyber security team, which is very disappointing for an organisation that handles such sensitive info.
This is so interesting and terrifying re having to sanitise documents, Jay. Thank you for sharing.
I looked into what you said and tested it out too and yep, deleted text still able to be retrieved. I’d say if that was the issue, the whole dept of Ed will have to consider their processes around this.
It is not the Department of Education. If you’re going to shoot, at least aim correctly.
I was unaware of this until today when the story came out.
To be honest I think it is a big deal and here is why….
One of my top students came out of Exam 1 and while discussing with friends realised that he had not noted the imaginary coefficients and hence the inability to proceed with conjugate root theorem.
He was beside himself for making such a silly error (tears and disbelief)… one that clearly was due to the enormous pressure of sitting the first question of an exam he had prepared so heavily for and placed so much weight on. (Knowing this student I know he knows how to do that question).
Now were he to get the question the day before, and have a quick look at it – this error would not have been made. If you are asking him whether he thinks it is a beat up I would suggest he would argue strongly it’s not.
That is a fair point. (I’m puzzled by how your student would have then factorised the equation but that does not properly negate your point.)
Agree with Srsly VCAA. Marty’s blog fails to consider all the additional external factors impacting these kids as part of sitting the exam, and for this reason, I cannot agree that it’s a ‘beat up’. Perhaps he can read the article in The Age written by a year 12 student about this matter. It was incredibly well written and anyone who can read that article and walk away believing this is all a storm in a tea cup has no empathy whatsoever for these kids.
Thanks, Devon. I don’t understand your comment. What do all the additional external factors have to do with evaluating this factor?
Wow, I’m surprised this got media attention.
I guess this disproves the idea that VCAA has “spare” papers in the event of exam leaks, considering other exams had similar issues. But tbh people who saw the leak prior to doing the exam obviously had an unfair advantage so I do think it is a big issue, particularly for weaker students.
But it is what it is, VCAA believes doing things like giving everyone a mark for a question (they did this last year) or only changing numbers from an exam leak makes everything fair and happily ever after. Obviously not accounting for students who spent extra time on questions that everyone ended up getting a mark for or weak students who would have gotten the question correctly if they had seen it prior.
I think VCAA is just the most lazy and incompetent assessment authority in Australia, I don’t hear about other states having issues like this (maybe they do and just don’t receive as much media attention) and I am glad I no longer have anything to do with shitshow they call VCE or VCAA.
Thanks, Allan. I still don’t think this is a big deal in terms of the SM test fairness, although I accept that Srsly VCAA, above, has an argument.
As to how it happened, of course I have no idea. One document I got through FOI a while back contained an off-the-cuff suggestion (and excuse) that VCAA is way underfunded in comparison to NESA. I don’t know if that is true, but of course the more stretched people are, the more likely there is to be such screw-ups. (None of this relates to my concerns, the long-time awfulness of the curriculum and CAS and SACs and exams.)
The public service has been run down over many years – that is now generally accepted. My impression was that Vic was worse than NSW. That said, they say never waste a crisis. Key is what (if anything), Minister wants to do beyond cautersising political issue.
In terms of SACs, do you think the VCE system was fairer and better when they had midterm external exams rather than the GAT + higher emphasis on SACs?
I was also reading more, and I’ve changed my perspective I think for specailist maths it was way overblown by the media (but this isn’t necessarily bad because hopefully it will push for positive change). The reason I think this wasn’t a big issue is because the question was extremely standard, 99% of students who did textbook questions throughout the year would know how to do it with or without the leak. If you go to the Cambridge Textbook chapter 6F, the chapter is practically full of questions of that style (knowing how to solve polynomials with non-real coefficients). If this was some sort of separator question rather than a standard question with a widely known solution pathway to students who actually put in the work, then I would agree that it’s a big issue but it wasn’t.
The only thing that would adversely affect students, in my opinion, is if VCAA says “we’ll give everyone 3 marks” or “give everyone a derived study score” (this was a legitimate suggestion in one of the articles I read) to “ensure no student is disadvantaged”. Since derived study scores are largely based on cohort performance too, I think this will only unfairly advantage students from highly ranked schools and shit on the face of those who attend lower ranked schools. If my understanding of how this all works is correct.
100% on a final end of year exam. The only way to assess properly and to keep everybody sane beforehand.
On the exam thing “overblown by the media” leading to “a push for positive change”, I very very much doubt it. The sharks and their useful idiot commentators are not looking for positive change, and they are very unlikely to get it.
I plan/hope to write again on this tomorrow.
The key question is whether they had briefed the Minister. Ministers HATE surprises.
According to the Australian Report (paywalled link above), Carroll was alerted a couple weeks ago.
The Guardian now has a story with response from the Premier.
https://www.theguardian.com/australia-news/2024/nov/14/vce-exam-cheat-sheets-answers-releases-vcaa-premier-jacinta-allan
Thanks, RS. I’ve added the link to the post as well.
Also made it to ABC TV News.
Thanks, RS. Yes, clearly everywhere.
This time last year Gniel ran away to his promotion to acara. Leaving a burning building and also an exams unit that just went through redundancies. Redundancy positions he chose because why have exam staff when you can have a few more corporate staff to answer his emails.
So he leaves Kylie with this disaster, only the disaster has 10% less staff but an added exam stress. Kylie comes in this year and has tried from what all can see but you put a bare bones staff under even further pressure and a tightened timeline then what do you expect was going to happen?
When will this be seen as Steve’s fuckup? Another example of a mediocre man in education leaving a mess for a woman to clean up.
What redundancies? Why?
As a teacher who’s taught in average/below average scoring schools, it is very common for below or about average students to use conjugate root theorem on questions like this. Seeing this would have been a tremendous advantage to those students. The question was being shared in large student group chats in the week prior to the exam and my students came out of the exam and commented that they were surprised the exam leak was real.
I agree with marty that for most students the question was easy enough anyway.
Thanks, Teech.
I think it is very unfair that this question was leaked, since even students who know their content well may have been advantaged if they had seen it. Firstly, they would recognise that the conjugate root theorem won’t apply. However, those who then “verify” the factor in part a, without any knowledge of the exam leaks, may potentially proceed with polynomial long division, or equating coefficients, which is a far longer process than recognising the grouping, and this exam leak would have advantaged some students to give them a hint as to which method is most efficient.
I don’t understand why VCAA can’t avoid a question on the actual exam that is too similar to one on the practice ones. Is it laziness of the VCAA education people (write once, use many, just change a few integers), or something else? Another question: more philosophical – is it unfair to give an advantage to a student who follows the sample questions vs. those who ignore them? I’m not pushing any barrows here, just genuinely wondering. Also what % of students check out the sample questions? Is it high or low (I’m not a teacher, nor a mathematician, but a lowly GP who came across this site after reading The Age article!).
Hi, PS. Lowly GPs are always welcome.
There are differing opinions of the advantage given to students who had seen the accidentally released Specialist question. (I have zero thoughts about other subjects.) I am of the opinion that it is no big deal. Others here seem more concerned, although it is mixed. Susie O’Brien, of course, is selling it as extremely as she can, freaking out students, while trumpeting her great concern for the freaked out students. Standard tabloid stuff.
As to why VCAA kept the format of the SM question the same, it is only a guess but I would suggest there is a 0% chance that it was laziness. My guess is that VCAA discovered the screw-up so late in the process that they had very little time to adjust the exam(s). Composing another, and non-similar, question on the same topic would have been trivial; it would take five minutes. But clearly there was a reason VCAA decided they could not do that, something to do with the lateness of the hour and the cumbersomeness of the process.
> Composing another, and non-similar, question on the same topic would have been trivial; it would take five minutes
That would be true if only spec was affected. In reality, VCAA would have had to completely replace the first page of every section of every exam, and I can imagine that that would have used a lot of resources
There are two things one should never try and prove: the impossible and the obvious. I have broken my silence at Marty’s blogs not to prove, but to attempt to explain, one or the other of these two things (depending on your opinion): Why I think access to the leaked Specialist Mathematics Exam1 question provided a non-trivial advantage. I am doing this because my silence might be misunderstood. I have previously given my reasons in a radio interview with Ali Moore on 774 ABC Drive (Thursday 14 November ~ 4.20 pm):
1) The topic the question examines (complex numbers and subsets of the complex plane) is broad. An ‘uninformed’ student will have spread their preparation across the entire topic (with possible extra attention on parts they found challenging). Knowing which part of the topic was likely to be examined and how it would be examined gives the ‘informed’ student the advantage of focusing on a much smaller part of the topic, or at least spending more time preparing that smaller part than they otherwise might, relative to an ‘uninformed’ student. If students are preparing how to solve one of thirty (*) different possible types of question for a particular topic but some of those students know exactly which one of those thirty types it will be, it is naive and disingenuous to claim that the students with foreknowledge don’t have a significant and unfair advantage relative to the students who don’t.
(*) Metaphorical but probably not too far off the mark.
2) Efficient techniques for solving this type of question are not necessarily taught or seen in every single class across the state. An ‘informed’ student who may not have seen this type of question before the leak gets the opportunity to ask for help with it.
3) An ‘informed’ student would notice that the question has one more part worth 1 mark and might intelligently speculate on what that part might ask. It is entirely reasonable that the ‘unseen’ part could be correctly inferred.
4) There is almost certainly a psychological (and probable performance enhancing) benefit to seeing a question on an exam that you have specifically prepared for. Intangibles such as these are almost certain to exist but obviously cannot be quantified.
In the same radio interview, I made it very clear that I think:
a) the VCAA CEO and the VCAA Director of Curriculum [EDIT] have done an excellent job producing the 2024 mathematics exams, which I think are the best quality we’ve seen for decades, and
b) it would be a very great shame if either or both of these two are made a scapegoat for ‘Leakgate’.
I had also expressed these views to several journalists, along with my great disappointment that ‘Leakgate’ has completely overshadowed the very hard work by the VCAA Director of Curriculum [EDIT] and her team, in partnership with the Monash University Mathematicians, in implementing all of the recommendations of the Bennett Review and producing the high quality 2024 mathematics exams. These opinions were not quoted in the media reports but I’m not whining about that.
I have reasonable confidence that the “full independent root-and-branch” review ordered by Minister Ben Carroll will result in positive outcomes akin to those achieved from the Bennett Review (also ordered by the Minister). In anticipation of scapegoats, I have offered the following question to several journalists:
Given the numerous serious mistakes etc. across 2022 and 2023, if VCAA CEO Kylie White is forced to resign or falls on her own sword:
[EDIT]
One might reasonably argue that the numerous serious mistakes etc. across 2022 and 2023 were as bad if not worse than ‘Leakgate’. One can also easily argue that Kylie White and the Director of Curriculum have been powerful agents for positive change at the VCAA.
[EDIT]
As a final comment, I have wondered, following the “high level meeting” between Minister Carroll and the VCAA, whether Kylie White might have been given a not-to-be-deviated-from script to follow in her media interviews on Thursday.
If all this makes me a “shark”, I can live with that (in fact, I’m happy to be likened to a misunderstood apex predator). But I think referring to hard working journalists, who are simply doing their job, as “useful idiot commentators” is an insulting and unjustified low blow.
PS – Yes, I should have said ‘pathways’ not “pathway”. I’m happy to attach the comparison document I wrote shortly after giving my quote for all to enjoy (including laughing at), upon request.
Thanks, John.
I’ve deleted one passage and a few phrases (all marked [EDIT]), which contained a falsehood and was too personally attacking of someone.
(If you do not accept these edits, you can instruct me to delete your entire comment and I will do so. If you make future comments that contain such attacks or such gratuitous inaccuracy, about which you have been repeatedly warned, I will simply delete your comments.)
Susie is no innocent simply doing her job, and I don’t think you’ve helped things. I’ll leave it at that.
5) As a result of preparation that was focused on the leaked question, an ‘informed’ student will most likely answer the question more quickly than an ‘uninformed’ student and hence have more time to answer the remaining examination questions. (Not a lot of time, but when 90 seconds = 1 mark, I think this gives a non-trivial advantage)
Kylie has resigned. I think you can stop now.
It was *never* about Kylie resigning. You (should) know that. It was about her statement that the questions had been (sufficiently) changed and her statement (and the opinion of others) that there was no (trivial) advantage to students. The first statement was clearly false, and I have given reasons why I disagree with the second. I also believe that 5) is relevant to the process now being used to try and clean up this mess.
You’re proud of your role?
I resent the implication. It’s a cheap shot and you (should) know it.
Are you proud that it was Allan’s undeleted comment at your blog here:
that most likely alerted the media to the story? Blind Freddy could have seen what would happen.
It is not an implication and it is not a cheap shot: it is a statement of fact.
No, I am not proud that Allan’s comment may have started this. I felt and feel very guilty about it, and not just that.
And what makes you think I feel much different? We both know Kylie was not to blame but she would (and had to) be the one that took the blame. And she obviously had her own reasons for making the statements that she made last Thursday.
Because you haven’t previously said any different.
You were being used, it was obvious you were being used, you were an idiot for allowing yourself to be used, and you’ve previously made no indication that you have any awareness of any of this.
This was always going to explode. Allan’s comment didn’t matter. Your media appearances didn’t matter. But at least I wasn’t an idiotic pawn.
And you’re wrong: Kylie didn’t have to take the blame. That was a media crusade as much as anything.
You were the one that said “Whoops” and subsequently opined that any advantage was nothing more than trivial (with what I thought was a pretty weak reason).
Re: “Because you haven’t previously said any different.”
I have so. From my first posted comment:
” … in a radio interview with Ali Moore on 774 ABC Drive (Thursday 14 November ~ 4.20 pm) … I made it very clear that I think:
a) the VCAA CEO and the VCAA Director of Curriculum [EDIT] have done an excellent job producing the 2024 mathematics exams, which I think are the best quality we’ve seen for decades, and
b) it would be a very great shame if either or both of these two are made a scapegoat for ‘Leakgate’.
…. I had also expressed these views to several journalists, along with my great disappointment that ‘Leakgate’ has completely overshadowed the very hard work by the VCAA Director of Curriculum [EDIT] and her team, in partnership with the Monash University Mathematicians, in implementing all of the recommendations of the Bennett Review and producing the high quality 2024 mathematics exams. These opinions were not quoted in the media reports but I’m not whining about that.”
My quotes objectively disagreed with Kylie’s claims that the questions had been (sufficiently) changed and that there was no (trivial) advantage to students. The first claim was demonstrably false and I gave reasons why I disagreed with the second claim (and the VCAA itself has since also disagreed with both claims).
First you said (by inference) that I used the media, and now you’re saying (explicitly) that the media used me. What next? We used each other?
PS – Re: “If you make future comments that contain such attacks or such gratuitous inaccuracy …”
I honestly don’t know what that comment is based on (but I suppose the [EDIT]s prove it). But I’m an idiot (how sweet it is) so what would I know.
I won’t be responding any further at this or any other blog, so have at it.
I’m curious what your thoughts are on VCAA’s “Anomalous Grade Process” (see https://www.vcaa.vic.edu.au/assessment/vce-assessment/Pages/2024VCEExaminations.aspx)
My main thought is that I don’t care.
I assume this is some variation of the “psychometric analysis” stuff that VCAA standardly does, to evaluate how questions have “performed”, particularly if, for some reason, there is suspicion that the question and/or answering is screwy. It is really annoying when VCAA uses this stuff to argue that a buggered question is not buggered, but that doesn’t mean that the stuff doesn’t make intrinsic sense. I’m not the one to judge, but I assume it does.
When things go wrong, you do your best. I have no reason to doubt that VCAA will do their best here, and that their best will be pretty good.