Tony Gardiner did some hunting and found a document containing PISA’s released mathematics items for 2022. In a comment on our PISA non-post, Tony made some remarks about a triple of triangle questions contained in the document. The questions, each with the same triangle picture prompt, are below. The questions are followed by PISA’s discussion. The detailed marking rubric can also be found in the items document (pp 9-14). Note that PISA is a test for 15-year olds; as with all such tests, the questions are intended to range from easy to difficult.

*1. For the first item, students are asked to compute the percentage of blue triangles shown in the first four rows of the pattern. There are six blue triangles and 16 total triangles, so the percentage of blue triangles is 37.5% (6 ÷ 16 = 0.375). This is an easy item (Level 1a) and is intended to get students thinking about the pattern by employing a simple algorithm with all information shown.*

*2. The second item in the unit builds off the first item by again asking students to compute the percentage of blue triangles, but this time it is based on five rows of the pattern. Since the fifth row is not shown, students have to extend the pattern by one row to determine new values for the number of blue triangles and the total number of triangles. With five rows, the percentage of blue triangles is 40.0% (10 blue triangles ÷ 25 total triangles). *

*This item is intended to be easy and to get students thinking about extending the pattern beyond what is shown, but not extending the pattern so that it requires generalising. This is a Level 2 item, so it is slightly more difficult than the first item in the unit, possibly because it requires working with a part of the pattern that is not shown but is still an overall easy item for students.*

*3. [The third item] is the final item in this unit, and it builds off the previous two items to now generalise with the pattern. The task for the students is to evaluate a claim that the percentage of blue triangles in the pattern will always be less than 50% as more rows are added. Students have to select either “Yes” or “No” to indicate if the claim is or is not true, but then they also have to provide an explanation to support their selection. This is a reasoning item that requires students to analyse the pattern to recognise a relationship between the number of red and the number of blue triangles in each row, and then use that relationship to support their selection. *

*The correct selection is “Yes,” that the claim is true, and an acceptable explanation recognises that the number of red triangles in each row will always be greater than the number of blue triangles in each row. Note that students can phrase their response in terms of either the number of blue triangles being fewer or the number of red triangles being greater, as long as there is some language indicating that this relationship is true for every row. Partial-credit responses to this item generally either focus on just the first row, which contains only a red triangle, or do not clearly communicate that the relationship between the number of each color triangle applies to every row.*

I’m confused what the purpose of this blog post is? Is there a mistake in the above questions? Are you saying that those questions are too easy? Are you just documenting this, due to it being hard to find?

Read Tony’s comment, or ponder. There’s no mistake per se, and it’s not a WitCH, although I’m not thrilled with the questions. Tony thought the questions were at least worth a comment, and I think they’re worth considering. As Tony noted, if one is going to discuss PISA tables, it is worthwhile looking at what is tested.

The first two parts use a triangle situation essentially to pose a simple arithmetic question, to determine the percentage that one number represents of another (the intermediate step is to count something to get the numbers, incidentally here triangles in a pattern). The second part is just pattern extension. The third part is an insight question, but the first two parts I think direct the thinking of the pupil in a not very productive direction; I can imagine a student getting bogged down in the details. The real question is whether there are more red triangles than blue, and I think that if you leave the counting and percentage aspects out of it, there are a couple of insightful ways of seeing the answer (e.g. looking at the arrangement in each row, or looking at rhombi with a red triangle on top). There is an artificiality about the question that does not appeal to me and it is not clear to me exactly what it is telling me about the students who do and do not answer it correctly.

Thanks, Ed. My dislike of the questions is similar.

I guess the idea is for kids to do the arithmetic of the first two questions (with a calculator), the results of which then helps them in pondering the third question. But it is far from clear to me that this is what kids would have likely done. You don’t have to be great with percentages to intuit that the red fraction is pretty low, and then, as Tony noted, that leaves you with only one possible option for each question.

The third question, I also agree. My guess is that plenty of kids will have had a strong and accurate sense of why there were more blues, but getting the sense and words out would have been tough.

As a module of questions testing mathematical ability, anywhere let alone across the planet, I think the questions are pretty awful. And these are the questions PISA chose to show us.

OK, so question 3 can be guessed pretty much by looking at the options for Q1 and Q2.

The “explain” part is a decent question, but I’m not sure how helpful this is in comparing different national education standards.

Nice enough questions (if you make them non MCQs) but not sure they are helping to measure what PISA claims these tests measure.

A bit like Methods/Specialist Paper 2 examinations… (delete this line if you feel it misplaced)

Thanks, RF. I agree, but I think it is made worse by the marking rubric:

Partial CreditCode 1: Selects Yes and explanation is partially correct but incomplete.

• [Yes] Because the first row has only a red triangle.

• [Yes] There are no blue triangles in the first row.

• [Yes] There is one more red triangle than blue triangle. [Response does not

specify “in each row”. Compare to Code 2, dot point 3.]

• [Yes] Because red triangles are on the outside of each row and the blue

triangles stay inside.

I must be too long in this job; that looks like a normal marking guide to me!

Not saying it isn’t awful, just that it looks… “normal” for school mathematics/numeracy tests.

Manykids are gonna know pretty much what is going on, but they’re not going to get into words more than the partial credit answers. Which means, as a test of whatmathematicsthe kids know, the rubric sucks.I have never met a student who sat for a PISA test.

Reminds me of the many many questions both in classroom tests and in exams which boil down to answering using “what do they want you to respond with” (as was the mantra of VCE) as opposed to a measurement of their actual understanding…

And those partial credit answers to the third question are just symptomatic of the lack of direction the prior two give. Yuck. I actually think this is probably the biggest reason I really despised maths assessment early on – the questions were exactly like this.

When the results for TIMSS 2023 come out in 2024, will they get the same attention that the PISA results have been given recently?

Probably, and with the same lack of comprehension.

Saw an item that discussed an SAT multiple choice question in which all of the answers provided were wrong. Yet another disadvantage of MCQs.

Don’t be ridiculous.