We haven’t paid all that much attention to the California Mathematics Framework, except for noting Jo Boaler (and Keith Devlin) making an idiot of herself again (ditto Devlin). We’re too busy with the local clowns. Greg Ashman, however, has noted a remarkable new front in the war over CMF, and it is worth highlighting.
Stanford mathematician Brian Conrad has begun a new site, a home for commentary on the CMF. Conrad introduces his site as follows:
High school students and parents deserve transparent, honest information about math skills required to earn STEM degrees — including in data science. It cannot be gimmickry with courses that suffice for college admission but leave students mathematically unprepared for their desired goals at a higher-education institution.
The California Math Framework (CMF) must be transparent and accurate in the guidance it provides on college preparation. As Director of Undergraduate Studies for Math at Stanford since 2013, I felt a responsibility to look into these matters.
As well as providing links to commentary critical of the CMF (here, here and here), Conrad has posted two articles of his own. Conrad’s first article, written with mathematicians Rafe Mazzeo and Patrick Callahan, is titled Key Mathematical Ideas to Promote Student Success in Introductory University Courses in Quantitative Fields. The document is pretty much what it advertises to be. The authors surveyed university colleagues in STEM fields, asking their opinions of the key mathematical ideas to be taught in school in order to “promote student success in introductory university courses in quantitative fields”. It is a simple, clear and useful document.
The second article posted by Conrad is astonishing. It is titled Citation Misrepresentation in the California Math Framework. Conrad introduces the document as follows:
The current draft of the CMF is a 900+ page document that is the outcome of an 11-month revision by a 5-person writing team supervised by a 20-person oversight team. As a hefty document with a large number of citations, it gives the impression of being a well-researched and evidence-based proposal. Unfortunately, this impression is incorrect.
I read the entire CMF, as well as many of the papers cited within it. The CMF contains false or misleading descriptions of many citations from the literature in neuroscience, acceleration, de-tracking, assessments, and more. (I consulted with three experts in neuroscience about the papers in that field which seemed to be used in the CMF in a concerning way.) Often the original papers arrive at conclusions opposite those claimed in the CMF.
In his article, Conrad then documents the many citation misrepresentations he claims the CMF contains, together with critiquing blatant nonsense. It is amazing work, a powerful calling of CMF’s scholarly bluff.
UPDATE (05/10/23)
Brian Conrad has written a post-mortem article for The Atlantic (semi-paywalled).
A lecturer at University where I’m currently studying for my M.Teach warned us about using references to papers quoted by other authors. She said that it is wise to read the original paper in order to make sure that what is being claimed by the ‘other author’ is what has actually been concluded in the original paper. The lecturer said that often, “for various reasons”, what was currently claimed was not what was originally concluded!
Sound advice from your lecturer.
I would go further: it is *essential* to read the original paper.
Absolutely agree, Terry!
Unsurprisingly, this sort of thing is also happening in Victoria. Those employed to teach at a public school will probably be aware of the “HITS” (High Impact Teaching Strategies), and may have been given a booklet detailing these strategies along with references to literature which justifies the effectiveness of these strategies. School leaders talk about this document as showing what is “evidence based” teaching practice. When I have followed up on the references, the cited literature does not support the government’s claims. For example, differentiation (ie. a single teacher assigning different tasks to students of different abilities within a single class) is one of the supposed “high impact” strategies. However the cited paper is about the effectiveness of targeted intervention programs (removing students with particular learning needs from a class for short periods for targeted instruction).
Thanks, SRK. Yes, this referencing sleight of hand is endemic. Is the HITS something that I should be aiming to hit?
It’s the latest golden bullet from the DET. It’s not all bad and a lot is good. It’s mostly a re-brand of basic, common sense stuff.
https://www.education.vic.gov.au/school/teachers/teachingresources/practice/improve/Pages/hits.aspx
I don’t think it deserves to be hit, but the lemming acceptance of anything – SOP by most school leaders – without appropriate skepticism and due diligence always deserves a whack.
SRK, I’d be very interested in any other examples you might be able to provide so that I can follow this up further.
I’d be *very* happy if someone else did all the work. Then I can post and get all the glory.
HITS is based on the research of John Hattie at the University of Melbourne. In a nutshell, he analyses lots of meta-analyses in education and comes up with measures of the relative impacts of many aspects on student learning. A simple example of a meta-analysis is attached.
2019-meta-analysis
What does Hattie prove that is not trivial or false?
In my experience, statistical analysis often confirms views of people at the coal face. The formal data analysis provides supporting evidence for those views, which are often based on gut-feelings developed over years of experience. Sometimes, formal statistical analysis contradicts these views. So one might say that many of Hattie’s findings are obvious to experienced teachers, but perhaps less so to policy makers.
A long way of answering “nothing”.
It is not the purpose of statistical analysis to *prove* that a propositions is true or false. Rather the purpose is to assess the evidence to consider whether or not the data support the propostion.
Yeah yeah. So what evidence does Hattie have to support a proposition that is not trivial or false?
Hattie doesn’t need evidence. He has a minion of followers and a large number of them are school leaders.
I don’t need to read his research (sarcasm, btw) as my school leaders quote it at me regularly.
Do they quote the trivial bits or the false bits?
Depends which leader.
I would argue that the two sets are not mutually exclusive either. No examples come to mind, yet.
The key reference is Hattie, J. (2009). Visible learning. Routledge.
Keep in mind that Hattie’s work is, in effect, a summary of findings in the literature.
I cannot comment with authority on this work because I have never got to the bottom of his methods – which are controversial. I have considered setting aside some time to do this but not managed to do so. It would be an interesting project for me.
One of Hattie’s findings is that the effect of reducing class size on student achievement is not as dramatic as one might think (pp. 85-87). Some might find this surprising.
Not really (others are welcome to disagree). PISA data shows almost an inverse correlation between average class sizes and PISA rankings… but I’m not sure this is a causal relationship.
I always start by saying that we should have class sizes of 80. Once the outrage dies down and everybody agrees that this would be a bad thing, I then say that it sounds like everyone agrees that class size
matter. Quite a bit …
I’ll back personal experience against so-called educational ‘research’ any day of the week.
It is not published research I’m quoting, just a rather unusual observation.
I don’t think there is a valid conclusion to be drawn, but I still think it is interesting.
Furthermore, if you look at average teacher salaries (converted into AUD using the Big Mac Index, not the official exchange rate…) then countries with better paid teachers tend to perform worse in PISA.
Once the hype dies down, I could also point out that countries that perform best on PISA seem to
the role of the teacher. Salaries and class have different effects in different 
of school structure, so are (in my opinion) irrelevant on an international level.
For the thousandth time, PISA is shit.
To JF: My year 4 class had n=70 with one teacher. It was OK as far as I recall. Indeed we had an excellent teacher – with an international reputation as I learned later in life.
To TM:
Yes, for a number of reasons I have no doubt that many classes ‘back in the day’ were superior to classes of today.
Unrelated – Are educational ‘experts’ and the allied PD gravy train full of snake oil sellers etc. relatively recent phenomena?
Very good point. It’s a lot easier to get 70 kids in a class to learn something if they’re all sitting still and facing the front and paying attention and practising decent exercises and not expecting or demanding to be entertained and to have their hands held every two minutes. But it doesn’t mean you couldn’t still do it all better if there were just 40 kids or just 20.
proposition (rather than propositions).
Of course there is the possibility that data can disprove a universal hypothesis such as “All swans are white”.
And Karl Popper argued at some length (and yes, I have the original paper) that good theories forbid things from happening and that the more they forbid, the better the theory.
Hence: all swans are white is a good theory. It is testable, forbids a lot of things from happening and is what some may therefore call a “risky prediction”.
“Give teachers better resources and students will do better at school” is not a risky prediction.
Differentiated learning (HITS 10) could fall into this category, but I’m not completely convinced as yet.
I think I’ve heard people talking of this one. I think they suggested it was in the “false” category.
Unfortunately, Hattie’s methodology is flawed. This means that although I agree with some, but not all, of his conclusions, this is just luck.
John Hattie is Wrong
The main problem is that he average ‘effect sizes’ from studies with very different designs and of very different quality and then places them on ladder with the interventions that have the largest average effect sizes at the top. Good quality studies tend to have low effect sizes and so what Hattie is reporting is mainly the results of poor quality studies. If you want your intervention to appear at the top of Hattie’s ladder, run lots of poor quality studies.
And what is an intervention? In order to find enough studies to average, Hattie groups together quite different things. This is why you have research on something that is not differentiation as commonly understood by teachers being used to support differentiation.
I drill into a similar issue with England’s EEF Toolkit here.
The article that England’s Chartered College will not print
Thanks very much, Greg. I’ll return to not paying any attention to Hattie.
Another article on this topic is here:
https://mje.mcgill.ca/article/view/9475
It was this controversy that made me enthusiastic about wanting to get to the bottom of the issue.
Thanks, Terry. I was tempted to stop not paying attention to Hattie, but I’ll now continue.
In spite of the above criticisms, I still have an open mind on the work of John Hattie because I don’t know enough about the detail of his work, and meta-analysis in general, to make up my mind. Still, the above criticisms do arouse my suspicions and I would like to get to the bottom of this issue.
Applied statistical work almost always has limitations. Was the sample of data really a random sample? Usually not. Were the variables really normally distributed? Usually not. The question then becomes: Are the limitations of the analysis damning?
I would like to get to the bottom of the issue. On the one hand, the High Impact Strategies (HITS) are benign, as others have pointed out above. The strategies are good common sense, and the HITS documents are useful in getting teachers to consider the strategies and debate them in their schools. On the other hand, these strategies are promoted by the Victorian Department of Education and Training as being based on evidence, and a key part of the evidence is Hattie, J. (2009). “Visible learning”, Routledge. Sorting out the issue is important – at least for me.
So, when time permits …
The big, big problem is that Hattie did not engage the services of a professional statistician. This is a big, big problem in social sciences ‘research’ and inevitably leads to flawed and invalid findings. Unfortunately, this doesn’t stop others, such as the DET and school leaders, from embracing these findings as the “Holy Grail” (of education, in Hattie’s case) – The other big problem is that others do not understand (or do not want to understand) how to critically evaluate such ‘research’ and its findings. School leaders are always more than ready to buy snake oil, even when they’re slapped in the face with why it’s invalid.
Once I attended a statistics conference where the keynote presentation was given by a professor of statistics from an Australian university. He wanted to talk about his new book which offered a fresh approach to teaching first year university students about statistics. During the presentation he outlined his approach to one of the exercises in chapter 1 of his book. There ensued about 30 minutes of debate about his approach to this problem. Fascinating.
I agree that a big problem is that people who want to use statistical results do not understand the details. I found this when I worked as a cancer statistician. See attached.
2013-ANZJPH
What an amazing and important piece of scholarship by Brian Conrad.
Citation misrepresentation is rife:
https://www.nature.com/articles/420594a
Deeply troubling is when the paper
been read and, either deliberately or through stupidity, a false or misleading description of the citation is given. Which then compounds citation misrepresentation … It is nothing short of gross academic fraud.
No doubt Spin Boaler and friends will cry foul and accuse Conrad of all sorts of nastiness.
Hey, good news. This is nice.
Is there any chance we could now debate this part of the post?
“It cannot be gimmickry with courses that suffice for college admission but leave students mathematically unprepared for their desired goals at a higher-education institution.”
Or has that ship well and truly sailed, at least here in Victoria in non-IB schools?
I don’t think VCE mathematics is gimmickry. It’s insanity, but it’s not gimmickry.
I’d say it’s gimmickry with how much you’re expected to use the CAS. Most problems in exam 2 are trivial using CAS gimmicks and too time consuming to be feasible in the exam without using the CAS.
Thanks, Anon. No question that the inclusion of CAS means that knowing CAS tricks/gimmicks is important to doing well. But, I don’t think the pedagogical decision to include CAS is gimmickry; it’s crackpottery.
I was thinking more about what “suffices for college admission” and the “leave students mathematically unprepared” parts of the statement, but yes, the CAS is well and truly a product that fits into my definition of gimmick.
It seems to me that Australian universities adapt their expectations so that students who enter the university have a good chance of graduating. We can see this from afar. In my undergraduate days, you could not study a language at university unless you had studied that language throughout your school years. These days, to study a language at university, you do not need any prior experience. My Year 7 Latin is now taught in first year Latin at university. We see the same sorts of changes in mathematics. It is not necessary to have studied calculus at school to study engineering at university. Market forces prevail.
I don’t really see a problem with this. Most schools don’t offer latin, and it would be bad to stop students from learning latin in university just because they didn’t go to a school that offered it. Sure, some students may get into an engineering course without doing Methods or Specialist, but calculus II or whatever first year mathematics course they have to do will pretty much ensure they have the proper fundamentals for doing engineering.
Of course there is a problem.
If Latin was a prerequisite for anything, more school would offer it.
When Specialist ceased to be a prerequisite for many degrees, the enrolments dropped. It is, as Terry quite correctly pointed out, a simple product of supply-and-demand.
The only problem is that Specialist then tried to make itself more attractive to try to boost enrolments again. I doubt it will work, but we can wait and see.
Docio ergo bibo…
Very good, RF.
When Physics ceased to be a prerequisite for Medicine, Physics enrolments went into freefall (even more so than Mathematics enrolments in Queensland after external exams were introduced).
Universities have the greatest influence on numbers enrolled in a subject. (And yet they bleat and mewl about standards).
Universities had control over the exams at one time as well.
We both know how that ended.
With a whimper, not a bang.
So… we might need more wine. That was the solution Arthur Dent and Ford Prefect decided on from memory…
A colleague once said that economists in universities tend to be in favour of market forces – except when it comes to making Economics optional in a business degree.
@Anon: I was not suggesting that adapting to market forces is a problem. I was just using Latin as an example. These sorts of developments allow more flexibility in the education system. By and large, this is a good thing.
I recall meeting a professor from a major university in the UK who gave a beautiful lecture on pure mathematics about shuffling cards. However, he had made his reputation in demography. Over coffee, I asked him how he moved from demography to pure mathematics. He told me that his first degree was in classics, but he needed a job so he swapped to demography. Then he moved to pure mathematics.
It’s a good thing that people can move.
If Latin were a prerequisite …
And I had only consumed one glass by that stage… should have learned from my last grammatical error.
I don’t think those edutards knew what hit them when a real scholar went and read their entire 900 page document, pulled most of the references, etc. That’s part of being a Ph.D. Ability to read a lot of content and extract parts of it and etc.
Conrad recently reiterated some of these concerns in the Atlantic:
https://www.theatlantic.com/ideas/archive/2023/10/california-math-framework-algebra/675509/
Thanks, Javert. I saw that but haven’t had a chance yet to read it.