We were alerted to this one by Simon the Likeable. It is a question from the QCAA 2025 Mathematical Methods Paper 2, together with the marking guide. (It may more properly be a PoSWW, but every time someone throws one of these stats questions at me, my brain tries to flee my body and so I can never properly judge.)
Visited 612 times, 3 visit(s) today
The alternative hypothesis is one sided (H1: p > 0.75) therefore a one sided CI is required. Calculating a finite upper bound is wrong. The correct CI is
. Therefore the claim is reasonable. The marking scheme is definitely not!
The marking scheme is erroneous. One would reasonably expect that the official results for many students are therefore wrong. The question is worth 5 marks. Given what happened here: https://mathematicalcrap.com/2023/06/02/qcaa-shows-some-more-class/
it is reasonable to think that some students will have missed out on higher VTAC offers because of this significant error. It will be interesting to see whether 2025 QCAA has the same integrity as 2023 QCAA.
Wait. If the interval had turned out to be (64.8%, 75.1%) are we supposed to say that it would be “reasonable” to publicly and confidently claim that employee satisfaction is above 75% ?? Is that the only way to interpret this?
Surely the board member is unreasonable by inspection.
The use of a two sided CI (such as the one you propose or the one calculated in the marking guide) to make a decision on whether or not to reject H0 is only appropriate if H1 is two sided. In this question, H1 is not two sided so the use of any two sided CI to make a decision is wrong.
IF the question had been worded
“…. the proportion of the satisfied employees in the entire company is NOT 75%.”
then the calculation and use of a two sided CI to make a decision is valid. For the CI you propose, the decision would be to accept H0 at the 0.05 level of significance. This decision is made because the proportion under H0 lies inside the CI. (You might word your decision as “There is insufficient evidence to reject Ho at the
level of significance”).
I strongly doubt that using a CI to make a decision when H1 is one sided is part of the QCAA Maths Methods curriculum. The vetter(s) should have detected this during the exam writing process and advised either an appropriate modification to the question or a new question.
I don’t think you’re addressing Alex’s point.
If so, then Alex (or someone) will need to clarify his point, keeping in mind my response.
Or, you could reread the exam question and Alex’s succinct criticism.
I don’t think Alex’s “succinct criticism” gets anywhere near (or even far) to why the marking guide is totally wrong (and why the question itself is probably defective within the scope of the QCAA syllabus). Which is why I wanted clarification. (I interpreted Alex’s comment as asking a genuine question, and replied accordingly). In fact, by using a two sided CI Alex is making the same conceptual mistake as the marking guide.
I think Alex is asking a very clear and very pertinent question, and you are not directly responding to his question. You are not obliged to, but I think someone should.
OK.
In the context of a very different question where H1 is two sided and Alex’s and a decision is made using Alex’s CI:
“Are we supposed to say that it would be“reasonable” to publicly and confidently claim that employee satisfaction is above 75% ??”
No.
level of significance”.
You would say what I said in my reply: “There is insufficient evidence to reject Ho at the
“Is that the only way to interpret this?”
No. See above.
“Surely the board member is unreasonable by inspection.”
You’ve put words in the board members mouth (*). The board member is unreasonable if they “publicly and confidently claim that employee satisfaction is above 75%”. The board member is not unreasonable if they say “There is insufficient evidence to reject Ho at the
level of significance”.
(*) However, I accept the proposition that most board members would probably use those words. And I accept the proposition that most board members are unreasonable by inspection.
I tried.
So did I. Obviously I can’t see the “very clear and very pertinent question.” (Which is why I asked for the clarification). I’ll let someone else see it and answer it.
Sorry if I confused things with the questions. They were very much rhetorical.
The point is that you can’t confidently say “the proportion is above 75%” when the only data available says “the proportion could be as low as 64.8%”.
It doesn’t matter if the interval is (0.648, 0.729) or (0.648, 0.751) or (0.648, ∞). There’s no way to justify the claim.
I agree that a sensible, related claim regarding “greater than x%” would use a one-sided interval. But you don’t need any of that to evaluate this particular claim.
No, Alex, you didn’t confuse things.
If this were a VCE question and there had been a statement along the lines of:
This is not what was written.
What was written is NOWHERE CLOSE TO THIS.
I do not believe the question can be answered as it is currently written.
Correction: H0 is p = 0.75. H1 is p > 0.75.
The question as currently written can be answered (using a one sided CI). It certainly shouldn’t be worth 5 marks (2 marks is more reasonable). But it’s very unlikely that one sided CI’s are within the scope of the syllabus.
I have little confidence in my stats knowledge, but am trying to make sense of this. Am I correct in assuming that if you want to claim the proportion is greater than 75%, the null hypothesis would be that the proportion is equal to 75%? And that the p-value is then the Pr(sample mean >0.6886)? Which is obviously greater than 0.05?
Yes. Yes. Yes.
And to add the obvious punchlines:
And p > 0.05 therefore there is insufficient evidence to reject Ho at the
level of significance?
Yes.
Therefore the marking scheme is totally wrong?
Yes.
(If only the exam writer and vetter(s) had the same little confidence as you do, Damo. Then this egregious error would never have seen the light of day).
This has already been pointed out, but I think it could be done in a more direct manner.
If you want to use a confidence interval to reject a null hypothesis for a two tail test, this is (mostly) fair (and in VCE it is a valid reason, given in some textbooks, such as Cambridge).
However… since
and 
are not stated here, the use of a confidence interval cannot be assumed to be valid.
(One sided confidence intervals are not in VCE and I doubt they are in QCE).
One should always frame the hypotheses *before* collecting the data and the question should be framed accordingly.
Whenever I see public examinations in mathematics, I always go to the questions on probability and statistics first. This is where I expect to see errors in questions and answers.
In my experience most secondary teachers have never studied probability or statistics to any serious level or had much (any) practical experience in industry. And we see the end results in public examinations, and in the curriculum.
Thanks, Terry. Do you have specific comments on this question, other than the temporal order?
A firm is planning to conduct a staff satisfaction survey. The need for this was prompted by the statement made at a board meeting that there was concern about staff satisfaction in the firm. There are about 3000 of staff in this firm. A survey was sent to all staff by email and 500 replied by email.
(a) Do you think that the sample of 500 staff who replied is a random sample of staff? Explain your answer.
(b) Let us assume that the sample of 500 staff who replied is a random sample of staff. The sample data showed that 400 members of staff who responded agreed that they were satisfied with their employment. Find a 95% confidence interval for the proportion of staff in the firm who are satisfied with their employment.
(c) Imagine that you are in charge of this survey. Write a short paragraph that present the results to the members of the board who are not as numerate as you are. Include any limitations of the study.
======
Comments:
* Some might say that this is a bit simple. It is simple. But this is what I would hope that numerate high school students ought to be able to do, and feel like they had learned something useful in their study of mathematics at school.
* Parts (a) and (c) are the hardest to answer well.
* Interesting questions: Why 95%? Why not 80% or 93%? What is so sacred about 95%? But that is for another time.
Do you want to answer my question?
I’m really getting sick of this.
Sorry; I misunderstood; I was trying to re-word the question; mea culpa.
No, you were trying to write a different question, and one which would have been infinitely better, but this was not your assigned homework.
I really, really hate statistics, and Simon the Likeable owes me a Henschke or an equally attractive alternative for having me embark on this idiotic WitCH on this idiotic topic. But a couple comments.
(1) The immediate problem of a WitCH is to critique the question as posed.
You guys are screwing this up.
Yes, you can then naturally suggest how the question should have been posed, or what question you think should have been posed instead, including the possibility far afield from the given curriculum. But these come after you or someone has made very clear what, if anything, is wrong with the question as posed.
(2) This is a question about a two-sided confidence interval
Maybe the question should have been about a one-sided confidence interval, but it was not.
Maybe the question should have been framed as a hypothesis testing question, but it was not.
This is what we are told:
a) A large company surveys the satisfaction of 500 of its employees.
b) THEN, a “95% confidence interval” [0.648, UP] for the “proportion” of satisfied employees was calculated.
(It is clear that [0.648,UP] was intended, but one could (tendentiously) argue that [0.648,
) was or should have been a permitted interpretation.)
c) THEN, a boss wants to “use the survey results” to argue for > 75% satisfaction.
That’s it. Don’t add in your own stuff to critique the question as written.
I’ve long since forgotten what you’re supposed to ‘do’ for the stats part. But reading this:
1. Doesn’t it matter how many employees there are in total? What if there are exactly 500- then isn’t the confidence “interval” just a point? Whereas if there were 1 million employees, its a huge extrapolation so the confidence interval should be much wider? They say a “large” company, but I would have thought 500 employees was already pretty large (This might just be my lack of stats knowledge showing though)
2. “Reasonable” is difficult to define… I would think 75% is a reasonable claim if its at or below the actual proportion of the 500 people that were satisfied. (Again, without knowing the size of the company I’m not sure how you would have enough information to calculate this proportion). But in the solutions they evaluate the upper bound of the confidence interval- I don’t see how that is relevant to 75% being a “reasonable” claim.
3. If we were actually working for this company, we would already know what proportion of the 500 people said they were satisfied. Having to work this out from the lower bound of the confidence interval is not actually a “real-world scenario”, if thats what they were aiming for.
Thanks, aps. If the confidence interval is just a point then that point must be [0.648, 0.648].
I cannot understand any scenario, and any interpretation of “reasonable”, where a confidence interval going down to 0.648 can justify a claim of the statistics-ish truth of (as opposed to the statistics-ish possibility of) > 0.75.
A student who answered,
“The 95% confidence interval has a lower bound of 0.648 and so it is plausible that the percentage of satisfied employees could be as low as 64.8%. Therefore the board member’s claim is not reasonable.”,
should be rewarded with 5 marks. If they were, then fine. If they were not, then very not fine.
Given the official solution, one might think they were not.
The official solution is not something that should be promoted as an exemplar solution to this question.
Thank you, Anthony.
CIs are often badly explained even in documents where the authors should know better.
The meaning of a 95% CI is this. If the same sampling experiment were conducted a large number of times, and the CIs were calculated, then about 95% of the CIs would contain the unknown population parameter. The remaining CIs so obtained would not.
I used to do the following experiment when I had a large number of students in Statistics 101 (about 100). Use your calculator to generate a sample of 50 random numbers between 0 and 1. Then find a 95% CI for the population mean (- which we know is 0.5).
About 5% of my students calculated a 95% CI that did not contain 0.5.
Or more precisely and in the context of population proportions:
Given a set of n C% confidence intervals, the number that contain the unknown population proportion is a random variable X with a distribution
Binomial(C/100, n).
E(X) = nC/100.