A Real PISA Work

Andreas Schleicher, that is. Of PISA itself, we couldn’t give a stuff. But apparently Australia still sucks at it.

For anybody who cares, the reporting on the just released results is, in no particular order, here, here, here, here, here, here, here, here, here and here.

17 Replies to “A Real PISA Work”

  1. PISA is a bit like Trump: it is hard to ignore, and yet one remains astonished at its influence.

    I once worked really hard (at ICME10) to make sense of “mathematical literacy”, and totally failed (http://michel.delord.free.fr/tony.pdf). The materials are often(?) fine for classroom use. But how they can possibly be administered in different education systems, and then be marked sufficiently objectively to produce meaningful numbers totally defeats me. The tasks used mostly have very little to do with the headline notion of “mathematical literacy” (and are sometimes crassly illiterate – see “Walking” below). So it is hard to see what they tell us about the efficacy of school maths teaching. And yet, and yet.

    My conclusion (having watched Schleicher at work) is that PISA is a social/political engineering project, which has hijacked UNESCO to pursue a specific agenda in a way that politicians find seductive. It is hard to know what to make of a billion dollar industry that can generate examples like “M123: Walking” (for picture, see https://www.oecd.org/pisa/38709418.pdf):
    “The picture shows the footprints of a man walking.
    The pacelength P is the distance between the rear of two consecutive footprints.
    For men, the formula n/P =140 gives an approximate relationship between n and P, where
    n = number of steps per minute, and P = pacelength in metres.”
    Yet 15 years after its flaws were explained, they continues to present this as a useful example. (Think: if I halve/double my pace length, do I really halve/double my pace rate – and hence travel at quarter/four times the speed?)

    1. Thanks, Tony. Your Mathematical Literacy essay is great. And that Walking Problem is remarkably stupid. Even for PISA.

      What do you regard as Schleicher’s/PISA’s agenda or social/political engineering project? They clearly have something in mind, and something pretty screwy, but I haven’t tried to decipher it more than to determine that it’s pretty screwy. Is it simply cartoon “numeracy”, or maybe what you refer to in your article as “quantitative literacy”?

    2. Sorry Tony

      As a former public servant, I’m afraid it’s not so hard ….

      “to know what to make of a [MULTI -my insertion] billion dollar industry that can generate examples like”

      ….. most reports prepared by the big four consultancies.

      1. “However that doesn’t mean it’s all hogwash.”

        Of course it doesn’t imply this *logically*; so by all means pause (just once in life) to check. But – having done so several times – methinks JJ leans over too far backwards to try to be fair.

        There is one key question:
        Is (mathematical) literacy, or numeracy like *the arithmetic of fractions* (i.e. something concrete, that can be taught systematically over time?
        Or is it like, well, “literacy”, or “maturity”: that is, a by-product of something else, that comes as a bonus if that “something else” is done well.

        The statements that
        “Its data may correlate in some way with reality (however one defines that). The fact that Singapore is up there at the top and that Australia has been declining is in line with the lived experience of maths teachers.”
        do not help us to decide.

        But from here the answer is clear. It is perfectly possible to treat “literacy” and “numeracy” as the focus of instruction. Indeed, for “numeracy”, one can even develop a detailed, blow-by-blow curriculum to match.
        And with enough effort, one can even make it work – up to a point (especially if one recognises that this attempt to replace “maths” by “numeracy” is politically driven, and if one then decides to cash in on this to blur the distinction between “numeracy” and “elementary mathematics”). Indeed, this is almost exactly what we did in England 1997-2006 – and are still doing to some extent.

        But it is pure “cargo cult science” (https://calteches.library.caltech.edu/51/2/CargoCult.htm). To go down this road is dishonest/foolhardy and utterly misguided – even though it can (sort of) be made to work *up to a point*. In my judgement, tts outcome – even if done well – is neither mathematics, nor education.

        The English National Numeracy Project/Strategy was in many ways a miracle. It took primary teachers who had been forced into a postmodern madhouse, and – to their great relief – imposed *structure* and purpose. But the structure was that of “numeracy” – not mathematics. They taught kids to get answers – and they did it rather well.
        No mathematics education researcher seems to have had reason to “pin the tail on the donkey”, but it is hard to resist the conclusion that this huge national effort must have contributed significantly to the *massive* rise in England’s TIMSS scores *at age 10* (these had been poor in 1995 and 1999 – in line with other evidence; but they jumped more than any other country in 2003; and have risen consistently, if much more gradually, ever since on every subsequent TIMSS, in 2007, 2011, 2015, 2019).

        So why am I not convinced?
        For two reasons:
        (i) Having had five kids, I am always delighted if someone wins the school egg-and-spoon race, or scrapes(!) Grade 4 violin, at age 10. But such triumphs are of limited value unless they lead on to something more serious later on. The purpose of primary mathematics is to prepare for what happens next: for school maths to be useable, the serious stuff tends to be faced at age 11-14. Each of the five “successful” TIMSS 10 year old cohorts (2003-2019) went on to form the 14 year old cohort for TIMSS four years later; and four years later their “success” as 10 year olds had (almost) no discernible effect! Politicians and educators continued to celebrate the 10 year old egg-and-spoon triumphs while ignoring the fly in the ointment at age 14.
        (ii) The “improved scores” at age 10 were genuine. But they were *average* scores. And, when I looked more closely, it seemed to me that we gained lots of marks in areas that are less important (getting answers to simple, direct problems), and often did poorly on items that I judged to be diagnostic of “readiness to progress”. (If most students can manage 18 + 7 = ??, but cannot manage 18 + ?? = 25, they may be being encouraged to “get answers” using dead-end methods – such as counting on fingers – rather than mastering place value.)

        1. Thanks very much, Tony. It’s interesting that “numeracy” has content, up to a point, and that teaching numeracy can have results, up to (but only up to) a point. I don’t think “numeracy” in it’s Australian version even gets up to this lower point.

    1. I interpret “numeracy” as the ability to understand and work with numbers.
      (I interpret “applied mathematics” as solving the heat equation).

      ACARA interprets numeracy as “encompasses the knowledge, skills, behaviours and dispositions that students need to use mathematics in a wide range of situations. It involves students recognising and understanding the role of mathematics in the world and having the dispositions and capacities to use mathematical knowledge and skills purposefully.”

      The DET interprets numeracy as “the knowledge, skills, behaviours and dispositions that students need in order to use mathematics in a wide range of situations.”

      The NESA interprets numeracy as “using mathematical ideas effectively to participate in daily life and make sense of the world.”

  2. In my book, the fact that a term such as ‘mathematical literacy’ is not clear to maths teachers is enough to suggest that it is not particularly useful – however my guess is that there have never in human history been more definitional type ‘angels on the head of a pin’ exercises undertaken than currently.

    Cui bono is always the question to ask. Data is a big industry and people love rankings. The key is always to dig into the data as you have done and clearly the PISA data is sketchy based on your examples.

    However that doesn’t mean it’s all hogwash. Its data may correlate in some way with reality (however one defines that). The fact that Singapore is up there at the top and that Australia has been declining is in line with the lived experience of maths teachers.

    The question is can the energies generated by all this expected hoopla be channelled? Since Singapore appears to be doing well, perhaps we could copy more of Singapore. As a first, from what I’ve read here, we could just adopt their maths curriculum. And then perhaps some of their teacher training and textbooks? I don’t know what they’re like, but surely better than ours.

  3. Marty mischievously asked “What do you regard as Schleicher’s/PISA’s agenda or social/political engineering project?”

    Marty’s question is not easy to answer in a public forum with real confidence. However, each of us is free to formulate an informed impression. I encourage anyone who may be inclined to take PISA seriously to spend 20 minutes listening to
    https://www.ted.com/talks/andreas_schleicher_use_data_to_build_better_schools .
    Everything that matters is visible. The message is fluent. And convincing.
    As long as one does not pause to think.

    Andreas fits perfectly into the modern context of the “TED talk” – where expertise and competence are determined by short presentations (which may be polished, but whose foundations remain inscrutable – the precise opposite of the peer reviewed publication, on which modern science was based until recently) . His history is hard to pin down. (I probably could pin it down, since I know a very close relative who is a truly excellent mathematician and educator. However, I prefer not to press.) He is said to have “studied physics”, though there is no mention of his having graduated. But he claims a “Masters” from … Deakin! Enough said (even though I spent a happy semester in Deakin).
    Somewhere he clearly studied “rhetoric”.

    The first thing to decide while watching (and the answer is not obvious) is whether this is a wise man who understands why and how all those buzz-words matter; or whether this is a con-artist who has cleverly identified the politician’s (and the modern public’s) weak points, and targeted them relentlessly and persuasively. And selectively!

    If others make the effort and have a view, I would be happy to share other impressions. I have no inside information; but I have observed him in action – in the same restaurant, but never at the same table.

    1. Hi Tony – my tuppence-worth is that PISA is entirely in line with the modern world.

      On the basis of what you’ve written and Wikipedia and nothing more except being an ex-bureaucrat myself, he looks very much like a modern bureaucrat to me. While he appears to be academically qualified [Wikipedia: “He studied physics in Hamburg and then mathematics at Deakin University, where he graduated with a Master of Science degree in 1992.”], there is no mention of any educational qualifications or experience.

      More and more, it seems that those with power are further distanced from what they exercise power over (few ex-teachers in Parliament and few in the Dept Education etc etc). As you note, TED talks and consultants thrive in the modern world. Benchmarking (which he appears to have driven), became a bureaucratic fashion in the 90s. He is taking a technocratic approach, entirely in line with modern bureaucratic fashions. He has driven PISA’s expansion from 1997 I would guess – a signature program is excellent for your career. And if it builds his empire, all the better. He identifies the right supporters and stakeholders and cultivates those as any bureaucrat would. He has found a niche and it works very nicely for him.

      As for most bureaucrats, he has no skin in the game of actual learning.

      The way I found out what works best was actually teaching kids maths and seeing that without times tables, it’s really hard to do algebra and that direct instruction worked. And that computers didn’t contribute much to learning at all and were a terrible distraction etc etc. But only after trying in a very poor way to do something ‘different’. Fortunately an experienced colleague set me quickly right. And surprise, surprise, this has been the way maths has been taught for centuries. Add to that Dylan William’s observation that we don’t really know how learning occurs. So how are people to know whether what PISA tests is appropriate, except by listening to people who actually do the job – but that is of course not the modern way. And even then many maths teachers seem to believe in this stuff!

      So he’s on safe turf and probably believes in what he is doing and would not of course listen to those who think otherwise – modern senior bureaucrats are not known for their modesty.

    2. Hi, Tony. I’m not sure if my question was mischievous, but your replying with that Schleicher video certainly was. If I had been in the same restaurant with the guy, I doubt I would have been able to eat.

  4. I should also add a project I saw but was kept out of. The Secretary of the Department (a former and now again consultant) brought in another consultant without tender to process the department’s data through a complex model run by a 25 year old who as far as I know had no statistics qualification. The only person to object was given very deep frowns by senior management. The data (quality and purpose for which it had been collected) did not support being treated in this manner, but apart from those who actually worked with the data, who knew and who dared to ask? The Acting Manager (appointed to the position without a merit process) who oversaw the project was given a special Leadership award for basically brown-nosing the Secretary. And people used the results without any idea it was complete hogwash, no doubt following the usual Powerpoint presentation.

    The Secretary thought he was doing a good thing, but frank and fearless is very very long gone – he knew better and wasn’t the type to listen. It’s only those who really understand the underlying data who know and they aren’t listened to.

    That’s how the system works these days – PISA fits right in.

  5. The fact that this thread has “died a death” may indicate lack of interest. However, for potential nerds and teachers, I shall risk a couple of additional observations

    1. The “National Report” for the four education systems in the UK is a miracle of vacuous, repetitious prose (180 pages that say very little, and which refuse to draw even the simplest inferences). In particular:
    (i) the report contains not one single *example* (see 4. below);
    (ii) the report speaks only about the *average* total scores (where all test items are combined and then the scores averaged): in line with (i), they show no interest in any details.

    2. An “average total score” blurs all distinctions
    (i) between what was easy and what was hard;
    (ii) between where students succeeded and where they struggled; etc.

    3. I’m not sure I even understand what an “average total score” means for PISA – since
    (i) each student only answered a small fraction of the problems (<10%)
    (ii) some (unknown percentage) of the students were given computer controlled "adaptive" versions (where the next question posed depended on how one had answered previous questions).
    In such a setting, the whole idea of an "average score" is a calculational fiction – a useful, but crude, first order measure.

    4. The really interesting thing is to study the individual examples used – together with the success rates, or scores achieved by different countries. Only then can one use one's experience to identify which questions are really indicative of strength or weakness, and which countries perform well or poorly.

    I have stated before that PISA items are often interesting for classroom use – but not obviously for reliable inter-system comparison. So I share a url which I stumbled upon (but not mentioned in the UK report!):

    Click to access CY8_TST_PISA2022_MS_Released_MAT_Items_FINAL_IH.pdf

    5. I hope teachers find the released items useful.
    But close-ups always show wrinkles!
    To give just one example – namely the item "Triangular pattern".
    There is nothing wrong with this as an item (indeed, it may be quite useful). However:
    (a) it is pure play – and has ittle to do with the grandiose PISA definition of "mathematical literacy";
    (b) (i) the first two questions concern "the fraction of blue tiles" (at first in the given picture; then in the imagined "next row";
    (ii) the second item asks (T/F) whether, in any extended pattern, the fraction of blue tiles will always be less than a half.
    These are reasonable questions to ask. But no-one seems to have noticed that in each of the first two multiple choice examples, *only one* of the multiple choice options is less than a half!

    1. Thanks very much, Tony, if only for reminding me how much I loathe PISA. The triangle thing is pretty dumb. I’ll post on it later today.

      I don’t think this post died so much, as I never tried to give it life. I felt obligated to write something on the PISA release, but I had no energy to attempt to interpret the league tables or, as you have, to hunt down actual test items. But my dead post did give re-birth to your “Mathematical Literacy” article. That was worth it.

      I agree with you, that it is only by looking at the test items, of PISA or TIMSS or whatever, that one can get some sense of what is being tested or not, and what students in a given region tend to know or not. I also don’t understand why both PISA and TIMSS make it so hard to locate test items. I have included and/or written about specific test items a few times: TIMSS, TIMSS, PISA, PISA, PISA.

      1. Just to clarify: I thought the “Triangular pattern” example was rather good!

        My comment was meant to indicate (i) that it was “naked maths” rather than “math literacy”; (ii) that the truly interesting question “Will blue triangles always be < 50% of all triangles?" should have alerted someone to the fact that *only one* of the options given in the multiple choice bits was < 50%.

        I repeat: many PISA tasks deserve an airing. But not obviously as part of an international, inter-system comparative study.

        1. Hi Tony. I thought the scenario/prompt was good, but not much else. In any case, I’ll post it and people can discuss.

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 128 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here