Our first post concerns an error in the 2016 Mathematical Methods Exam 2 (year 12 in Victoria, Australia). It is not close to the silliest mathematics we’ve come across, and not even the silliest error to occur in a Methods exam. Indeed, most Methods exams are riddled with nonsense. For several reasons, however, whacking this particular error is a good way to begin: the error occurs in a recent and important exam; the error is pretty dumb; it took a special effort to make the error; and the subsequent handling of the error demonstrates the fundamental (lack of) character of the Victorian Curriculum and Assessment Authority.
The problem, first pointed out to us by teacher and friend John Kermond, is in Section B of the exam and concerns Question 3(h)(ii). This question relates to a probability distribution with “probability density function”
![\[ f(x) = \left\{\aligned &\frac{(210-x)e^{\frac{x-210}{20}}}{400} \qquad && 0\leqslant x \leqslant 210,\\ &0 && \text{elsewhere.} \endaligned\right.}\]](https://mathematicalcrap.com/wp-content/ql-cache/quicklatex.com-a7289281aa496a7cf618bd82731e4593_l3.svg "Rendered by QuickLaTeX.com")
Now, anyone with a good nose for calculus is going to be thinking “uh-oh”. It is a fundamental property of a PDF that the total integral (underlying area) should equal 1. But how are all those integrated powers of e going to cancel out? Well, they don’t. What has been defined is only approximately a PDF, with a total area of 
. (It is easy to calculate the area exactly using integration by parts.)
Below we’ll discuss the absurdity of handing students a non-PDF, but back to the exam question. 3(h)(ii) asks the students to find the median of the “probability distribution”, correct to two decimal places. Since the question makes no sense for a non-PDF, of course the VCAA have shot themself in the foot. However, we can still attempt to make some sense of the question, which is when we discover that the VCAA has also shot themself in the other foot.
The median m of a probability distribution is the half-way point. So, in the integration context here we want the m for which
a) 
As such, this question was intended to be just another CAS exercise, and so both trivial and pointless: push the button, write down the answer and on to the next question. The problem is, the median can also be determined by the equation
b) 
or by the equation
c) 
And, since our function is only approximately a PDF, these three equations necessarily give three different answers: to the demanded two decimal places the answers are respectively 176.45, 176.43 and 176.44. Doh!
What to make of this? There are two obvious questions.
1. How did the VCAA end up with a PDF which isn’t a PDF?
It would be astonishing if all of the exam’s writers and checkers failed to notice the integral was not 1. It is even more astonishing if all the writers-checkers recognised and were comfortable with a non-PDF. Especially since the VCAA can be notoriously, absurdly fussy about the form and precision of answers (see below).
2. How was the error in 3(h)(ii) not detected?
It should have been routine for this mistake to have been detected and corrected with any decent vetting. Yes, we all make mistakes. Mistakes in very important exams, however, should not be so common, and the VCAA seems to make a habit of it.
OK, so the VCAA stuffed up. It happens. What happened next? That’s where the VCAA’s arrogance and cowardice shine bright for all to see. The one and only sentence in the Examiners’ Report that remotely addresses the error is:
“As [the] function f is a close approximation of the [???] probability density function, answers to the nearest integer were accepted”.
The wording is clumsy, and no concession has been made that the best (and uniquely correct) answer is “The question is stuffed up”, but it seems that solutions to all of a), b) and c) above were accepted. The problem, however, isn’t with the grading of the question.
It is perhaps too much to expect an insufferably arrogant VCAA to apologise, to express anything approximating regret for yet another error. But how could the VCAA fail to understand the necessity of a clear and explicit acknowledgement of the error? Apart from demonstrating total gutlessness, it is fundamentally unprofessional. How are students and teachers, especially new teachers, supposed to read the exam question and report? How are students and teachers supposed to approach such questions in the future? Are they still expected to employ the precise definitions that they have learned? Or, are they supposed to now presume that near enough is good enough?
For a pompous finale, the Examiners’ Report follows up by snarking that, in writing the integral for the PDF, “The dx was often missing from students’ working”. One would have thought that the examiners might have dispensed with their finely honed prissiness for that one paragraph. But no. For some clowns it’s never the wrong time to whine about a missing dx.
UPDATE (16 June): In the comments below, Terry Mills has made the excellent point that the prior question on the exam is similarly problematic. 3(h)(i) asks students to calculate the mean of the probability distribution, which would normally be calculated as 
. For our non-PDF, however, we should should normalise by dividing by 
. To the demanded two decimal places, that changes the answer from the Examiners’ Report’s 170.01 to 170.06.
UPDATE (05/07/22): The examination report was updated on 18/07/20, and now (mostly) fesses up to the nonsense in 3(h)(ii). There is still no submission for the parallel nonsense in 3(h)(i).
Thanks for your article, Marty. It articulates many of my own feelings and the feelings of many teachers I have discussed this question with. Such an obvious error on the exam is unforgivable. But it is the Assessors Report that upsets me the most. The comments about this question in the Report are (in my view) irresponsible and mealy-mouthed. Nowhere is there an acknowledgement that an error was made – it reads as though ‘almost a pdf’ was deliberate. The comments are a complete whitewash in my view), no doubt to avoid any potential *ahem* class-action. I’ll tell you how some inexperienced but diligent teachers will interpret these comments:
“I better give my students a function that is a close approximation to a pdf as practice in case a similar question appears in the future ….”
It deeply troubles me that no mention is made of students who said that the median does not exist because the function is not a pdf (prefaced with some appropriate calculations – so as to earn the ‘method mark’!) nor is it ever stated in the Report that such an answer is correct and acceptable.
More mathemathicalcrap.com, please !
This is a most worthy service to learners and teachers!
The examiners report seems like post election spin
The first part of this question asks students to find the mean value.
Perhaps this should be calculated as:
(\int_{0}^{210} xf(x) dx) / (\int_{0}^{210} f(x) dx) .
Terry, that is an excellent point. I’ll update the post.
This has just been drawn to my attention by Marty.
The function f CAN be turned into a probability density function (pdf) by changing just one symbol. If 0 \le x \le 210 is replaced by -\infty < x \le 210 (or even more simply x \le 210) then it is a pdf – it is the density of 210 – a Gamma(2,20) random variable. So the "best" answer for the median would be 210 – the median of a Gamma(2,20) random variable, giving 176.433, and the "best" answer for the mean doesn't need a calculator – it is 210 – 2 * 20 = 170.
It would be a good exercise for second year probability to give them the VCAA published f and ask them the simplest way to turn this into a pdf.
Mind you – it is a pretty silly random variable because the Gamma(2,20) distribution is that of the waiting time for the second arrival in a Poisson process of rate 20 per unit time. Since we get negative answers possible for 210 – this random variable, perhaps the examiners had time travel in mind!
Thanks very much, Tim. This suggests that possibly the examiners grabbed a genuine pdf and deliberately chopped it off. If so, I think it is easy to answer *why* they would have chopped it off: as far as I am aware, the Methods (and Specialist) curriculum includes no treatment of, or even mention of, improper integrals. (The exception is the normal distribution, where the magic buttons do all the work.) Of course, independent of the specific nonsense discussed here, this is a very silly curriculum decision.
Anyway, if that is indeed the reasoning of how the pseudo-pdf was created, it in no way justifies the decision to do so. As for the examiners employing time travel, yes that would perhaps then be the silliest modelling aspect to have appeared in a Methods exam. But, only perhaps: the competition is pretty fierce.
The earth turns slowly but the ox is patient.
As a consequence of the error in the unit for momentum on the 2018 Specialist Maths Exam 1 Q 6 and the lack of acknowledgement in the Examiners Report of this error, I have attempted since the end of Feb 2019 to get an explanation from VCAA on why a number of errors – particularly this pdf error – never got mentioned in the Reports.
After several months of emails, I have currently (27 June 2019) been told by the Examinations Unit – again – that:
“I met with David Leigh-Lancaster about your concerns about the 2016 Maths Methods question. He is not aware of any error in relation to a missing PDF/diagram.
Neither was any error reported to us by teachers or students.”
I was then met with a condescending
“If you are after a Mathematics perspective about the question, can I suggest that you call or email David Leigh-Lancaster.”
My reply to all this was met with: “I will need to refer your question to David for further advice.”
So after – apparently – speaking to the VCAA Mathematics Manager several times, the Examination Unit needs to speak with him about it some more. What the ….?
This is what happens when you try to get transparency from VCAA.
And it’s not just VCAA – there are commercial companies that sell solutions to the VCE exams that have deliberately ignored any mention of this error – presumably because the writers have a conflict of interest ….
Thanks very much, John. It is great that you are pursuing this. A few thoughts.
Clearly the person in the Examinations Unit interpreted “PDF” to mean a file/graphic rather than a probability distribution. That is understandable, since they are presumably a general administrator rather than a semi-demi-mathematician. DLL, on the other hand, knows damn well what a PDF is (up to ±0.0003).
So, what happened? Did DLL honestly believe that the query was about a missing diagram? Or, did he deceitfully play along with the examination unit’s misunderstanding? If not, is the examinations unit now deceitfully playing with you? To quote from the response you received:
[DLL] is not aware of any error in relation to a missing PDF/diagram. (Emphasis added).
Is this just lazy or random specificity, or is it consciously deceptive specificity? You are obviously querying whether there is an error on the question, so why not state clearly whether or not DLL is aware of any such error?
To continue the quote:
Neither was any error reported to us by teachers or students.
If true that is astonishing, and every Head of Maths should then seriously consider their professional responsibilities and what they entail. But, even if true, it does not preclude the possibility that the examinations unit and/or DLL is aware of such an error.
God knows what is really going on here. It is too easy to read a non-existent deep meaning in short responses. But, it seems to me that the VCAA (= EU + DLL) is either being incredibly slippery or incredibly obtuse. Or both.
A further update (7 July 2019):
Apparently the Examination Development Panel has now been asked to provide further advice (I don’t know why DLL is suddenly silent on the matter). Furthermore, I’ve been told – yet again – that
“… there was no issue with this question raised with us by teachers through either the exam feedback survey or correspondence, until [mine].”
This is despite over-whelming evidence in the Examiners Report from weasel statements like:
“As function f is a close approximation to the probability density function …”
“… answers to the nearest integer were accepted.” even though the question explicitly asked for answers “… correct to two decimal places.”
This is despite the fact that I myself specifically raised this issue on the exam feedback survey (which is therefore a broken process at best).
Are we really to believe that nobody but myself (who got ignored) raised this issue on the exam feedback survey?
Are we really to believe that no Head of Department or teacher raised this issue with VCAA?
Are we to believe that no mathematical organisation raised this issue with VCAA? (Actually I can believe that one, because I have seen commercial solutions to this exam that completely ignore the issue).
Are we really to believe that none of the dozens of Assessors raised this issue with the Chief Assessor, either during the ‘training day’ or whilst marking this exam?
Actually, it would be interesting to hear from someone who was an Assessor of Exam 2 that year …. Hold on, what’s that you say? …. We won’t? ….. Because all the Assessors are forced to sign confidentiality clauses and aren’t allowed to discuss the marking process ….?
Well now, isn’t that mighty convenient. Cue the X-Files music ….
Thanks very much, John. I don’t know what the “Examination Development Panel” is, or how it might relate to DLL and/or the Examiners. But two things:
*) As you have noted, the claim that no teacher reported the error through the formal feedback system is false. So, how did that false statement come to be? This alone seems to me worthy of pursuit.
**) Is the claim really that *no one* in the VCAA other than the writers of the Examiners’ Report knew of the error? If so, that points to a fundamental flaw in administration. If not, then who knew what and when?
Marty, it never occurred to me that pdf might get interpreted as a graphic by the Examinations Unit!! But that makes perfect sense, particularly with the media attention on problems VCAA have had with some of their exam graphics over the last few years.
That DLL would play along with this interpretation is, lamentably, a very reasonable hypothesis (and a charitable one, given the alternative hypothesis). He must think it’s Pretty Damn Funny.
I can only agree with your conclusions. I’ll post an update when I get a reply. In the meantime, I’m going to start investigating options for escalating the matter outside of VCAA.
Update: I got a reply over 2 months ago that the “examination development panel” would be asked to “provide further advice” …. No reply since. It looks like everyone is too busy trying to make the current mathematics ‘consultation’ process
(see Marty’s latest blog: https://mathematicalcrap.com/2019/09/07/the-vcaas-mathematical-reasoning/)
look legitimate ….
Related to this error, I’ve only just noticed that the Cambridge 2019 Maths Methods Checkpoints has this same question (Q238 I think, from memory) and their ‘solution’ seems to be a direct copy from the ‘solutions’ of a well-known commercial organisation – an answer ‘correct’ to 2 dp and no mention of the error (I doubt this would have happened while Neil Duncan was curating Checkpoints).
I’ve just had a reply.
The 2016 Examination Panel Chair (who has since retired) and the Chief Assessor have been consulted. Apparently both said that the issue with the probability distribution function was raised before the Assessor Training Meeting and it (along with all questions) was discussed in depth at the Assessor Training Meeting. Apparently there were no issues for the students.
My response was to ask (directly quoting from Marty’s blog since he put it so well):
How can VCAA fail to understand the necessity of a clear and explicit acknowledgement of the error? This is fundamentally unprofessional. How are students and teachers, especially new teachers, supposed to read the exam question and report? How are students and teachers supposed to approach such questions in the future? Are they still expected to employ the precise definitions that they have learned? Or, are they supposed to now presume that near enough is good enough? The in-depth discussion at the Assessor Training Meeting is not reflected in the Examination Report – did that discussion admit that there was a blatant error? An amendment to the 2016 Exam 2 Examination Report that addresses these issues is clearly required. I would appreciate further clarification on this.
I am awaiting a response, which I hope to get in about 2 months.
Still waiting – nearly 1 year after first asking the question. And not one peep from the guy who heads VCAA’s Maths unit.
Final update:
VCAA have amended the 2016 Exam 2 Examination Report. The following comment has been appended to Q3 (h)(ii):
“As f is not a probability density function but a close approximation, answers to the nearest integer were accepted.”
Unfortunately it’s not the mea culpa we all might have hoped for, but at least it does explicitly admit that f was not a pdf (implicitly admitting an error was made) and it now more clearly explains the real reason why answers to the nearest integer were accepted. This is particularly valuable for inexperienced teachers accessing the report.
A proper re-write of the feedback would have been preferable. An explicit admission of the error together with an apology was always too much to hope for.
I’ve heard there will be amendments to several more examination reports (Specialist) in the coming days, so watch out for a few more “amended [fill in the date] 2020” (2020 might be a watershed year, bringing the most ever report amendments in a single year).
It’s small progress, but nevertheless it is progress. Now all we need is a particular organisation that, among other things, claims to foster the promotion of mathematics education, to amend its commercial solutions so as to reflect the sentiments in this report.
So long and thanks for all the fish.
JF, I searched far and wide to find the post referencing the 2016 Methods Exam 2, and luckily stumbled upon this one before throwing in the towel.
Thank goodness the Assessment Report has been amended (noticed the amendment this morning when scrolling through the Methods Examination page), however an amendment 4 years later is an absolute travesty.
4 years to add a sentence stating that f wasn’t a pdf? Seriously?
“Who are these clowns?” – https://www.youtube.com/watch?v=KTuPiHwUFGo
If I wanted to assess whether students could calculate a parameter of a given pdf, I would use a simple pdf. This should involve simple calculus. My aim would not be to test the student’s knowledge of calculus – this would be assessed elsewhere.
Indeed. But this is Exam 2 and so buttons must be mindlessly pushed. (Which is why the question was *mindlessly* written and vetted). Furthermore, if simple calculus was involved the VCAA exam writer would not look learned. (This way we get to see *exactly* how learned those responsible at VCAA for this farce really are).
And Steve, what you’re suggesting is common sense – the meaning of this phrase is foreign to VCAA.
@Steve: It’s common knowledge who the clowns are and who the biggest clown 54.0466° N, 2.8007° W is (I know your question is rhetorical). Clowns are meant to bring a smile to your face, not make you blood-boilingly angry.
From what I know of this, it took nearly a year from the initial email request asking for an amendment to the Report before VCAA (kind-of) acknowledged there was an error. During this time, you would not believe (although maybe you would) the extraordinary ducking and weaving by VCAA and the number of follow-up emails needed to continue pursuing this – there were explicit denials by VCAA that there was any error, multiple assertions that no previous feedback of an error had been given, multiple assertions that it was only an *opinion* that there was any error etc.
I expect similar prevarication and recalcitrance by some elements of VCAA should anyone decide to pursue the error discussed here: https://mathematicalcrap.com/2020/09/08/witch-43-period-piece/#comment-4439
Terry, you’re way too polite.