VCAA Has Deloitte to Learn

Last year, as seemingly occurs every year, there were serious errors on the VCE mathematics exams. This was reported on at the time by the Age, on the basis of a teacher’s complaint and a comment on this blog. The errors were reported on again last month by the Herald Sun, on the basis of a parliamentary submission. I was pleased to see these reports although they were not my doing. I did not notify any reporters of the exam errors and no reporter had asked me for comment until after the recent Herald Sun and Guardian articles on this year’s General Mathematics exam errors.

As I routinely do, I quickly blogged on some (but not all) of the 2022 errors (here, here, here and here), but other than that I had been as quiet as possible about the whole thing, even on this blog. For a reason, and until now.

This post will tell what I know of the background and prehistory of this nonsense, and in particular what I know of VCAA’s review(s) of the 2022 exams. I know a decent amount, although there is much more that I do not know and wish to know. VCAA has many questions to answer.

 

1. The Reporting

The Age report last year contained motherhood statements from VCAA but nothing of substance, no explicit denial of errors on the exams (and of course no confessions). The recent Herald Sun report, however, did. The report made reference to an external review of the VCE mathematics exams:

It’s understood the review, conducted by Deloitte using outside maths experts, found the language could have been more clear, but did not find serious errors that impacted on students. [emphasis added]

Seemingly on the basis of this external review, VCAA rejected claims of errors on the exams, with the Herald Sun quoting a VCAA spokesperson:

“The VCAA has confirmed that the questions did not present errors.” [emphasis added]

Following on from the Herald Sun report, local ABC radio had a news bulletin citing a VCAA spokesperson:

… the Victorian Curriculum Assessment Authority … [has] confirmed there were no errors in the 2022 exam [sic]. [emphasis added]

So, two separate media reports, quoting a VCAA spokesperson to the effect that there were no errors on the 2022 exams.

It is unclear how VCAA came to issue such a definitive statement, which, beyond being trivial to falsify, it is very clear VCAA knows to be false. How can we be so sure that VCAA knows this? Because of the Herald Sun remark on the Deloitte review, that the review “did not find serious errors that impacted on students”. One only makes such a weird and slippery claim if there is something they’re not telling.

It is not even clear what the statement on the Deloitte review means, but whatever it means, slippery as it is, the statement is still best thought of as false. Were there “serious errors”? Well, one would consider that mathematical falsehood, and unstated and misstated hypotheses reach the threshold of “serious”, and of course these errors impacted on students. Perhaps VCAA meant to imply that it was the impact that was not serious but, particularly given the QCAA incident, it is highly implausible and such a claim cannot be taken on faith. VCAA would seem to have a very difficult job trying to prove this and there is no evidence that they have properly tried.

I shall write a lot on VCAA’s external review, which is at the heart of this nonsense. In particular, I shall return to the suggestion of Deloitte “using outside maths experts”. With any reasonable definition of “using” and “maths experts”, this is also very likely false.

 

2. The Prehistory

In May last year, Burkard Polster and I requested a meeting with Suzy Chandler, then VCAA’s Executive Director of Curriculum, and Michael MacNeill, VCAA’s Curriculum Manager of Mathematics. The purpose of the meeting was to discuss the VCE mathematics exams, particularly for Mathematical Methods and Specialist Mathematics. Our concern was not simply about the presence, every year, of serious errors, but also generally about the poor quality of the exams and the exam reports.

Suzy and Michael appeared happy, and were at least willing, to meet, and the meeting occurred the following week. I think it was overall a good meeting, which I would characterise as tough but cordial. For an hour, Burkard and I pointed out, with many supporting examples, deficiencies in the past exams. To their credit, Suzy and Michael basically took it. They did not argue back, did not attempt to claim the bad was not bad. Suzy made some small attempts to defend the writing and vetting panels, but she didn’t try hard and it didn’t wash: the past exams were compelling evidence that whatever was in place wasn’t working. There were frustrating aspects, with Suzy referring to current VCAA processes as if they were determined by holy writ, but overall it felt as if Burkard and I were being taken seriously.

The meeting didn’t resolve anything but it seemed to be a solid basis upon which to keep talking, and I felt it ended on good, sincere terms. Follow up emails of genuine thanks were sent in both directions. Michael, who was new in his role, particularly seemed open to critique of the current state of affairs. We left on friendly enough terms for Michael to contact me soon after, twice, seeking my input on unrelated aspects of VCE mathematics.

The terms of engagement changed in November, soon after the 2022 exams. Predictably, these exams contained serious errors along with the general awfulness. The complex number error in particular, essentially a repeat of the 2021 complex error, was astonishing and inexcusable. Burkard emailed Jason Smallwood, who had replaced Suzy as Acting Executive Director, and Michael MacNeill:

Dear Jason and Michael,

After last year’s meeting with Suzy Chandler and Michael on the VCE mathematics exams, Suzy replied that the VCAA would be happy to talk further if required. Recent events have made clear that it is now required. 

As you are surely both aware, this year’s VCE exams contained serious errors, including errors beyond those reported. Beyond the errors, the structure and writing of the exams was deeply flawed, in exactly the manner that Marty and I flagged last year. It is clear that VCAA’s process for the writing and vetting of the mathematics exams is simply not working. It is clear that, for whatever reason, properly competent and attentive mathematicians have been excluded. 

Last year, Marty and I offered to vet the exams for free, or to arrange for other serious mathematicians to do so. This was immediately dismissed by Suzy.* This dismissal is no longer tenable, if it ever was. However it is to be done, VCAA needs to undertake a [thorough], clean appraisal of the VCE mathematics exams, and preferably VCE mathematics more broadly.

Marty and I would be happy to meet with you both, if you see a genuine purpose. But we are no longer happy to sit by and wait to see what happens. If our only option is to take our concerns elsewhere, we shall do so.

Kind Regards, Burkard

* (Burkard and I could have been clearer on this point. Suzy was fine with mathematicians vetting the exams, but she indicated mathematicians would have to apply to do so, like everyone else. We replied that there was no way either of us would apply to be, or encourage another mathematician to apply to be, simply another person on the team.)

Soon after, Smallwood replied to me and Burkard, requesting that we put our concerns in writing, and indicating VCAA’s intention to conduct an external review:

I write to you requesting that you (and Marty) provide all your concerns in writing, identifying the serious errors, and your concerns regarding the structure, development and vetting processes. 

It is our intention at the VCAA to provide your written concerns to an external provider for review and analysis, and to provide us with a confidential report on any findings that we may need to consider for the development of future Mathematics examinations. …

VCAA’s decision to respond with an external review was understandable and reasonable, and was possibly also motivated by the Age report, and by teacher complaints to and about the VCAA Examination Unit. But this formalisation also effectively closed the avenue to discussion. In particular, Burkard and I have heard nothing further from Michael. It is only conjecture, but it feels as if Michael has been instructed to have no further contact with us.

 

3. The Critique and the Five Major Errors

From the outset, Burkard and I were suspicious of VCAA’s external review. There was no indication of who the “external provider” might be. There was also no indication what the terms of reference might be and Smallwood’s emphasis on “process” was troubling. Yes the processes had to be considered, but this would logically come after a proper analysis of the poor outcomes of these processes, which goes way beyond the serious errors. The poor outcomes were also my and Burkard’s stated focus and were all that we were qualified to critique.

Perhaps most troubling was Smallwood’s emphasis that the subsequent report would be confidential, with me and Burkard only to receive a summary:

I will let you know once the review has been completed and provide you with an overview of the report once completed.

It was difficult to see why such a report couldn’t, and shouldn’t, be open to the public, at least in very large part.

Still, it was VCAA’s game. So, despite our strong concerns, we played along. In December, Burkard and I wrote and then emailed to VCAA a detailed critique of the 2022 VCAA Methods and Specialist exams:

The introduction to our critique itemised our concerns with VCAA’s review. Then, our critique proper began with what Burkard and I referred to as major errors on the 2022 exams. At that stage, and until February, VCAA had released no exam reports or summary solutions or anything, nothing at all to indicate what VCAA regarded as the correct answers or how the questions would be graded. Subsequently, after the appearance of the exam reports in February, Burkard and I put together a short summary document of the five “major errors”, which included the full questions and the reports’ summaries:

In short, the exam reports were completely silent on the errors. It is difficult to view this silence as anything but a conscious decision, as a refusal to acknowledge any serious issues.

 

4. VCAA’s Internal Review

Meanwhile, in another part of town, VCAA was conducting an internal review of the 2022 exams. In November, a teacher had complained to the VCAA Examination Unit about the four Specialist Mathematics questions in error. Finally, in March, the teacher received a reply:

VCAA had quickly confirmed receipt of our critique. Then, in February, VCAA confirmed that the exam reports, which had just appeared, would form part of the external review. At that stage, however, we had heard nothing further on the nature of the review. Then, in March, three months after we had sent VCAA our critique, Smallwood emailed us:

We wish to advise you that we have appointed Deloitte to conduct an independent examination into the process undertaken by VCAA to develop the 2022 VCE Mathematics Exams (specifically the General (Further) Mathematics, Mathematical Methods, and Specialist Mathematics exams).

The choice of Deloitte struck us as absurd (and absurdly expensive). Sure, Deloitte would have a few mathsy guys around, but maths education is not exactly Deloitte’s strong suit. One would think they’d be hard pressed to find people well qualified for such a review. But, Smallwood’s emphasis on “process” was the key:

The objective of the review is to ascertain whether VCAA acted in accordance with its policies and procedures in relation to the writing of the exams. As part of the review the accuracy of the individual exam questions themselves will be reviewed also. Such review may consider similar processes undertaken for other VCAA examinations and whether the 2022 exam results were in line with results of prior years.

So, the exam questions would be reviewed, but “also”: this was declared to be secondary. The clear focus of the review was to be “whether VCAA acted in accordance with its policies and procedures”. Easy stuff for Deloitte, but way, way off the proper point.

Smallwood also reiterated VCAA’s concerns for confidentiality, but in a more determined tone:

The VCAA may discuss or share a summary of the review outcome with you, whilst upholding its existing policies relating to the confidentiality of the VCAA processes and the individuals involved in the exam development. 

Burkard and I were not thrilled, to put it mildly, to have waited three months for such nonsense. A couple days later, after confirming that our critique had been forwarded to Deloitte, Burkard replied in detail:

Dear Jason, 

Thank you for confirming that our critique has been provided to Deloitte. 

It is concerning, however, that the objective of the review as you have stated it does not appear to address properly or fully the issues that Marty and I detailed in our critique. Our focus was on the mathematical content and the quality of the exam writing, and thus the procedure for writing and vetting exams, as well as the communication to the general public regarding of errors in marking and setting of examinations. In particular, our concern with what we regard as the substandard quality of the exams extended much further than the continual occurrence of blatant error, egregious though this is. It is difficult to see how a review of VCAA’s compliance with its own policies and procedures could address these much broader concerns in anything other than a peripheral fashion.

Perhaps our concerns with the review are unfounded, and the review has a clear mandate to address Marty’s and my much more extensive and substantive issues with the exams. We cannot be assured that this is the case, however, without more information.

To this end, so that Marty and I can properly understand the nature of the review Deloitte is undertaking, I request that you please forward:

a) The terms of reference of the review, as provided to Deloitte;

b) A clear indication of “[VCAA’s] policies and procedures in relation to the writing of the exams” that Deloitte’s review is intended to address, i.e., details of or references to specific policies and procedures;

c) A list of any other documents provided to Deloitte for the purposes of their review.

Thank you very much. 

Kind Regards, Burkard

In May, over two months later, Smallwood (now Executive Director) replied:

Dear Burkard,

I apologise that it has taken me a while to respond to your email sent on the 22 March. 

An overview of the Deloitte engagement scope has already been provided to you in my previous email dated 20 March 2023. I can confirm that nothing further will be provided to you or Marty regarding the scope of the review.

And so it went on:

As you may appreciate, information and documents held by VCAA are confidential and pursuant to our policies cannot be shared with third parties. 

VCAA has engaged Deloitte for its own purposes with the scope of work being broader than only addressing your concerns. The VCAA has taken your allegations seriously, but the outcome of the review (other than a specific summary which may be shared with third parties) remain confidential to the VCAA. 

It is VCAA’s view that the process of developing the exam content is the most appropriate component of the Deloitte review, to ensure compliance with VCAA policies and guidelines in relation to the development of the 2022 Mathematics Exams; and to identify any improvement opportunities for future exam preparation. 

It was only with the final sentence that Smallwood addressed dealing with the substance, dealing with the actual exams:

As part of the review, the VCAA and Deloitte may consider the use of another maths expert to assess the accuracy and quality of question content.

Burkard and I had no idea what “maths expert” meant, nor what “another” meant: where was the first “maths expert”? In any case, we had no further interest in trying to engage with Smallwood. Burkard thanked Smallwood for his reply and requested the email contact for VCAA’s CEO, Stephen Gniel, which Smallwood promptly supplied.

A few days later, Burkard emailed Gniel:

Dear Stephen,

You will of course be aware that VCAA has appointed Deloitte to conduct a review related to the VCE mathematics examinations. You may also be aware that Marty Ross and I are concerned about what appears to us to be the misguided objective of this review. In brief, and to quote from Jason Smallwood’s email to us from 20 March,

“The objective of the review is to ascertain whether VCAA acted in accordance with its policies and procedures in relation to the writing of the exams.”[emphasis added]

This focus upon adherence to policy appears to marginalise Marty’s and my central concerns, that the exams are specifically mathematically flawed and are of generally poor quality; to us, this is the issue of paramount importance, and anything else will logically follow.

On 22 March, I emailed Jason, indicating Marty’s and my concerns, and requesting clarification. It was very disappointing to then have to wait six weeks for Jason’s three-paragraph reply, which not only failed to provide any meaningful clarification, but also flagged VCAA giving even less consideration to Marty’s and my concerns than we had been led to believe.

Jason’s email of 4 May made explicit to “confirm” that there is no point in my requesting again any further information on the nature of Deloitte’s review. There are a few points, however, that Marty and I consider it necessary to have documented:

1) On 16 November last year, I emailed Jason and Michael MacNeill, requesting a (further) discussion with me and Marty of the VCE mathematics exams. Jason’s reply to our request, on November 27, was to ask that we put our concerns in writing, and to inform us of VCAA’s intention to have these concerns forwarded to an external provider for review and analysis. We did as Jason requested. It should be clear that we worked extraordinarily hard on the requested critique.

As such, although Jason is of course correct that “VCAA has engaged Deloitte for its own purposes”, it is important to recognise that Marty and I were informed that the review was undertaken on the basis of our concerns, to be framed upon the critique VCAA requested from us, and was VCAA’s response to us to a direct request to have further discussions with VCAA.

2) Jason indicated that VCAA and Deloitte “may consider the use of another maths expert” (emphases added). The relegation of Marty’s and my concerns to an afterthought, for VCAA-Deloitte to possibly engage some unspecified “maths expert”, goes to the heart of Marty’s and my concerns with the objective of the review.

3) We are of course aware that VCAA has rights and obligations regarding confidentiality, as determined by law as well as codes of professional conduct. Marty and I do not know the extent to which VCAA is permitted or required to withhold, or to disclose, certain information about the nature of Deloitte’s review and, when it appears, the review itself. We accept that some information may have to remain confidential out of concern for third parties. We do not accept without argument, however, the stifling requirement for confidentially that VCAA appears to be claiming without provision of a justification.

In his email of May 4, Jason notes that 

“the outcome of the review (other than a specific summary which may be shared with third parties) remain confidential to the VCAA. [emphasis in the original]”

In our critique, Marty and I noted our belief that VCAA has a problem with openness and candour. VCAA’s stated approach to Deloitte’s review has done nothing so far to dissuade us from this belief.

Marty and I remain committed to do what we can to facilitate the much-needed improvement of the standards of VCE mathematics exams. We await the outcome of Deloitte’s review.

Kind Regards, Burkard

Gniel replied (to Burkard only) a couple weeks later:

We assure you that we take your feedback seriously and have taken steps to ensure that the review process adequately addresses the potential mathematical flaws and overall quality concerns raised.

… It is essential that your input is valued, and we aim to ensure more timely and thorough responses moving forward

… We will make every effort to clarify the confidentiality requirements and work towards finding the right balance between transparency and protecting the interests of third parties.

… Thank you for bringing these matters to our attention. We assure you that your concerns are being taken seriously, and we are actively working towards a resolution that aligns with your expectations.

Gniel’s tone was significantly more considerate and conciliatory (to Burkard). The question was whether Gniel’s words were more than words.

 

6. VCAA’s Summary of the External Review(s)

At some time, no later than September 7 (the date of the Herald Sun report), and possibly months earlier, VCAA received Deloitte’s report. On October 13, over a month later, Smallwood emailed me and Burkard, getting Burkard’s name wrong, notifying us that VCAA’s external review had concluded, and attaching a summary of the review’s findings:

The review and its summary are absurd in all the ways that it appeared they would be. The summary is contemptuously absent of any detail, and Deloitte’s review appeared to be as misfocussed on policy as VCAA indicated it would be:

During the review the following activities were conducted by Deloitte:

• Discussions with 15 participants involved in the 2022 Maths Exam development process.

• Review of relevant documents and assessment of the writing and vetting process of the 2022 Maths exams against the relevant VCAA policy and guides

• Comparison of the publicly available information relating to the VCAA exam development process to other state bodies

Deloitte apparently did not have a word to say about the mathematical soundness or otherwise of the actual exam questions. VCAA, however, had other “maths experts” for this afterthought task:

In addition to the above scope, the VCAA sought the opinion of two independent state Education bodies on a sample of questions across each of the 2022 Maths Exams, including the questions raised in your letter dated 20 December 2022. Each body was asked to complete either all or a sample of questions including an explanation of their answer and provide commentary on the quality of questions, appropriateness to VCE level, whether there is any mathematical ambiguity and general language and grammar.

That is, the other “maths expert” turns out to be whomever happened to be around at the “independent state Education bodies”, who then reviewed “all or a sample of” the exam questions. Then,

The independent state bodies found that there were no major mathematical errors within any of the questions, however identified some room for improvement in the language and grammar used and level of marks provided per question. [emphasis added]

So, “no major mathematical errors”. But we are provided no clue as to who considered which questions, nor what they wrote about them.

Close to a year of waiting, to receive this vacuous, whitewashing nonsense.

 

7. What Now?

Burkard replied to Smallwood, acknowledging receipt. He then emailed Gniel, concluding with ten questions and requesting a proper reply within two weeks:

Dear Stephen,

On Friday, Marty and I received Jason’s email and his attached summary letter of VCAA’s review. We are not at all surprised at the outcome since the review appears to be misdirected and deficient in the precise manner that we flagged in our submission and in our email replies to both you and Jason. Nevertheless, we are certainly disappointed and frustrated. You also personally assured us that VCAA would seek the right balance between confidentiality and transparency: it is impossible for us to view Jason’s letter as remotely fulfilling this assurance.

The examination questions that Marty and I flagged as having major errors, which were a key reason for our having written to VCAA, are deeply flawed. One may choose to label the flaws in different terms but the questions are deeply flawed nonetheless. This is purely an issue of mathematics and it is not up for debate: we have detailed the flaws and no competent mathematician, no “maths expert”, as Jason phrased it, could possibly consider these examination questions remotely acceptable. The examination questions could not possibly have been attempted by the students in a consistent manner, and they could not possibly have been graded in a consistent manner, fair to all. These simple truths cannot be whitewashed with an, at best, tangentially relevant policy review by Deloitte, and it cannot be whitewashed with VCAA appealing to the unprovided arguments and conclusions of two “state education bodies”. Indeed, the failure of Deloitte and the state education bodies to properly censure any of these examination questions, if true, only points to the failure of VCAA’s review.

In the interest of at least somewhat addressing the current lack of transparency, I have listed below a number of concrete questions, the answers to which Marty and I would consider natural and obvious to have been included in Jason’s summary, and to which we would appreciate prompt answers. We regard these as straight, factual questions, which should be straightforward to answer and for which we cannot imagine issues of confidentiality being relevant.  

Marty and I trust that, after waiting so many months to receive so little, you can ensure that at least these factual questions will be addressed in a prompt and sincere manner. If it is to be longer than two weeks, we would appreciate a reasonable and proper estimate of how long will be required.  

Kind Regards, Burkard

*************************************************

1) When did VCAA receive the Deloitte’s report and the responses from the two state education bodies?

2) Precisely which examination questions were the two state education bodies presented, and which questions did each state education body “complete”?

3) What were VCAA’s grading schemes for the five questions that Marty and I labelled as containing major mathematical errors?

4) For the five questions that Marty and I labelled as containing major mathematical errors, what were the arguments and conclusions of the two state education bodies?

5) Did either Deloitte or the state education bodies review VCAA’s grading schemes and/or the examination reports? On the questions Marty and I labelled as containing major mathematical errors, what was the reviewers’ assessment on the adequacy of the grading schemes and of the examination reports? 

6)  For the seven questions that Marty and I labelled as containing minor mathematical errors, what were the arguments and conclusions of the two state education bodies?

7) For the sixteen questions that Marty and I labelled as being poorly conceived, what were the arguments and conclusions of the two state education bodies?

8) Which questions did each of the two state education bodies find to be “ambiguous”, and how was this ambiguity described?

9) Which questions did each of the two state education bodies find to be deficient in “general language and grammar”, and how was this deficiency described?

10) What were Deloitte’s recommendations, and which of these recommendations are being implemented by VCAA?

Two weeks have passed. Gniel has responded to Burkard’s email but has not responded to our request for substantive information about VCAA’s review. It’s not clear that it matters. VCAA had their chance and they blew it. 

For well nigh a year Burkard and I were silent on all this, hoping to provide VCAA the most space, the best opportunity to take these issues seriously and to try to get it right. We were silent even when there was compelling evidence that VCAA was not getting it right and seemingly had no interest in getting it right. Clearly, there is no longer any purpose in our being quiet. And thus this post. And this, and this.

Obviously things are now in flux, and it is unclear what will happen. But something has to change.