NAPLAN’s Numeracy Test

NAPLAN has been much in the news of late, with moves for the tests to go online while simultaneously there have been loud calls to scrap the tests entirely. And, the 2018 NAPLAN tests have just come and gone. We plan to write about all this in the near future, and in particular we’re curious to see if the 2018 tests can top 2017’s clanger. For now, we offer a little, telling tidbit about ACARA.

In 2014, we submitted FOI applications to ACARA for the 2012-2014 NAPLAN Numeracy tests. This followed a long and bizarre but ultimately successful battle to formally obtain the 2008-2011 tests, now available here: some, though far from all, of the ludicrous details of that battle are documented here. Our requests for the 2012-2014 papers were denied by ACARA, then denied again after ACARA’s internal “review”. They were denied once more by the Office of the Australian Information Commissioner. We won’t go into OAIC’s decision here, except to state that we regard it as industry-capture idiocy. We lacked the energy and the lawyers, however, to pursue the matter further.

Here, we shall highlight one hilarious component of ACARA’s reasoning. As part of their review of our FOI applications, ACARA was obliged under the FOI Act to consider the public interest arguments for or against disclosure. In summary, ACARA’s FOI officer evaluated the arguments for disclosure as follows:

  • Promoting the objects of the FOI Act — 1/10
  • Informing a debate on a matter of public importance — 1/10
  • Promoting effective oversight of public expenditure — 0/10

Yes, the scoring is farcical and self-serving, but let’s ignore that.

ACARA’s FOI officer went on to “total” the public interest arguments in favour of disclosure. They obtained a “total” of 2/10.

Seriously.

We then requested an internal review, pointing out, along with much other nonsense, ACARA’s FOI officer’s dodgy scoring and dodgier arithmetic. The internal “review” was undertaken by ACARA’s CEO. His “revised” scoring was as follows:

  • Promoting the objects of the FOI Act — 1/10
  • Informing a debate on a matter of public importance — 1/10
  • Promoting effective oversight of public expenditure — 0/10

And his revised total? Once again, 2/10.

Seriously.

These are the clowns in charge of testing Australian students’ numeracy.

The Wild and Woolly West

So, much crap, so little time.

OK, after a long period of dealing with other stuff (shovelled on by the evil Mathologer), we’re back. There’s a big backlog, and in particular we’re working hard to find an ounce of sense in Gonski, Version N. But, first, there’s a competition to finalise, and an associated educational authority to whack.

It appears that no one pays any attention to Western Australian maths education. This, as we’ll see, is a good thing. (Alternatively, no one gave a stuff about the prize, in which case, fair enough.) So, congratulations to Number 8, who wins by default. We’ll be in touch.

A reminder, the competition was to point out the nonsense in Part 1 and Part 2 of the 2017 West Australian Mathematics Applications Exam. As with our previous challenge, this competition was inspired by one specifically awful question. The particular Applications question, however, should not distract from the Exam’s very general clunkiness. The entire Exam is amateurish, as one rabble rouser expressed it, plagued by clumsy mathematics and ambiguous phrasing.

The heavy lifting in the critique below is due to the semi-anonymous Charlie. So, a very big thanks to Charlie, specifically for his detailed remarks on the Exam, and more generally for not being willing to accept that a third rate exam is simply par for WA’s course. (Hello, Victorians? Anyone there? Hello?)

We’ll get to the singularly awful question, and the singularly awful formal response, below.  First, however, we’ll provide a sample of some of the examiners’ lesser crimes. None of these other crimes are hanging offences, though some slapping wouldn’t go astray, and a couple questions probably warrant a whipping. We won’t go into much detail; clarification can be gained by referring to the Exam papers. We also don’t address the Exam as a whole in terms of the adequacy of its coverage of the Applications curriculum, though there are apparently significant issues in this regard.

Question 1, the phrasing is confusing in parts, as was noted by Number 8. It would have been worthwhile for the examiners to explicitly state that the first term Tn corresponds to n = 1. Also, when asking for the first term ( i.e. the first Tn) less than 500, it would have helped to have specifically asked for the corresponding index n (which is naturally obtained as a first step), and then for Tn.

Question 2(b)(ii), it is a little slack to claim that “an allocation of delivery drivers cannot me made yet”.

Question 5 deals with a survey, a table of answers to a yes-or-no question. It grates to have the responses to the question recorded as “agree” or “disagree”. In part (b), students are asked to identify the explanatory variable; the answer, however, depends upon what one is seeking to explain.

Question 6(a) is utterly ridiculous. The choice for the student is either to embark upon a laborious and calculator-free and who-gives-a-damn process of guess-and-check-and-cross-your-fingers, or to solve the travelling salesman problem.

Question 8(b) is clumsily and critically ambiguous, since it is not stated whether the payments are to be made at the beginning or the end of each quarter.

Question 10 involves some pretty clunky modelling. In particular, starting with 400 bacteria in a dish is out by an order of magnitude, or six.

Question 11(d) is worded appallingly. We are told that one of two projects will require an extra three hours to compete. Then we have to choose which project “for the completion time to be at a minimum”. Yes, one can make sense of the question, but it requires a monster of an effort.

Question 14 is fundamentally ambiguous, in the same manner as Question 8(b); it is not indicated whether the repayments are to be made at the beginning or end of each period.

 

That was good fun, especially the slapping. But now it’s time for the main event:

QUESTION 3

Question 3(a) concerns a planar graph with five faces and five vertices, A, B, C, D and E:

What is wrong with this question? As evinced by the graphs pictured above, pretty much everything.

As pointed out by Number 8, Part (i) can only be answered (by Euler’s formula) if the graph is assumed to be connected. In Part (ii), it is weird and it turns out to be seriously misleading to refer to “the” planar graph. Next, the Hamiltonian cycle requested in Part (iii) is only guaranteed to exist if the graph is assumed to be both connected and simple. Finally, in Part (iv) any answer is possible, and the answer is not uniquely determined even if we restrict to simple connected graphs.

It is evident that the entire question is a mess. Most of the question, though not Part (iv), is rescued by assuming that any graph should be connected and simple. There is also no reason, however, why students should feel free or obliged to make that assumption. Moreover, any such reading of 3(a) would implicitly conflict with 3(b), which explicitly refers to a “simple connected graph” three times.

So, how has WA’s Schools Curriculum and Standards Authority subsequently addressed their mess? This is where things get ridiculous, and seriously annoying. The only publicly available document discussing the Exam is the summary report, which is an accomplished exercise in saying nothing. Specifically, this report makes no mention of the many issues with the Exam. More generally, the summary report says little of substance or of interest to anyone, amounting to little more than admin box-ticking.

The first document that addresses Question 3 in detail is the non-public graders’ Marking Key. The Key begins with the declaration that it is “an explicit statement about [sic] what the examining panel expect of candidates when they respond to particular examination items.” [emphasis added].

What, then, are the explicit expectations in the Marking Key for Question 3(a)? In Part (i) Euler’s formula is applied without comment. For Part (ii) a sample graph is drawn, which happens to be simple, connected and semi-Eulerian; no indication is given that other, fundamentally different graphs are also possible. For Part (iii), a Hamiltonian cycle is indicated for the sample graph, with no indication that non-Hamiltonian graphs are also possible. In Part (iv), it is declared that “the” graph is semi-Eulerian, with no indication that the graph may non-Eulerian (even if simple and connected) or Eulerian.

In summary, the Marking Key makes not a single mention of graphs being simple or connected, nor what can happen if they are not. If the writers of the Key were properly aware of these issues they have given no such indication. The Key merely confirms and compounds the errors in the Exam.

Question 3 is also addressed, absurdly, in the non-public Examination Report. The Report notes that Question 3(a) failed to explicitly state “the” graph was assumed to be connected, but that “candidates made this assumption [but not the assumption of simplicity?]; particularly as they were required to determine a Hamiltonian cycle for the graph in part (iii)”. That’s it.

Well, yes, it’s obviously the students’ responsibility to look ahead at later parts of a question to determine what they should assume in earlier parts. Moreover, if they do so, they may, unlike the examiners, make proper and sufficient assumptions. Moreover, they may observe that no such assumptions are sufficient for the final part of the question.

Of course what almost certainly happened is that the students constructed the simplest graph they could, which in the vast majority of cases would have been simple and connected and Hamiltonian. But we simply cannot tell how many students were puzzled, or for how long, or whether they had to start from scratch after drawing a “wrong” graph.

In any case, the presumed fact that most (but not all) students were unaffected does not alter the other facts: that the examiners bollocksed the question; that they then bollocksed the Marking Key; that they then bollocksed the explanation of both. And, that SCSA‘s disingenuous and incompetent ass-covering is conveniently hidden from public view.

The SCSA is not the most dishonest or inept educational authority in Australia, and their Applications Exam is not the worst of 2017. But one has to hand it to them, they’ve given it the old college try.

A Lack of Moral Authority

The Victorian Minister for Education has announced that the state’s senior school curriculum will undergo a review. The stated focus of the review is to consider whether “there should be a more explicit requirement for students to meet minimum standards of literacy and numeracy …“. The review appears to be strongly supported by industry, with a representative of the Australian Industry Group noting that “many companies complained school leavers made mistakes in spelling and grammar, and could not do basic maths“.

Dumb and dumber.

First, let’s note that Victorian schools have 12 years (plus prep) to teach the 3 Rs. That works out to 4 years (plus prep/3) per R, yet somehow it’s not working. Somehow the standards are sufficiently low that senior students can scale an exhausting mountain of assignments and exams, and still too many students come out lacking basic skills.

Secondly, the Minister has determined that the review will be conducted by the VCAA, the body already responsible for Victorian education.

If the definition of insanity is doing the same thing over and over and expecting different results, then the definition of insane governance is expecting the arrogant clown factory responsible for years of educational idiocy to have any willingness or ability to fix it.

Don’t Go West, Young Man

No, Burkard’s and my new book has nothing to do with going West. A signed copy of our book is, however, first prize in our new Spot the Exam Error(s) Competition. (Some information on our previous exam competition can be found here.)

We spend a fair amount of our blog time hammering Victoria’s school curriculum authority and their silly exams. Earlier this year, a colleague indicated that there were perhaps similar issues out West. We then had a long email exchange with the semi-anonymous Charlie, who pointed out many issues with the 2017 West Australian Mathematics Applications Exam. (Here is part 1 and part 2. WA’s Applications corresponds to Victoria’s Further Mathematics.)

Following our discussion with Charlie, we sent a short but strong letter to WA’s School Curriculum Standards Authority, criticising one specific question and suggesting our (and some others’) general concerns. Their polite fobbing off indicated that our comments regarding the particular question “will be looked into”. Generally on the exam, they responded: “Feedback from teachers and candidates indicates the examination was well received and that the examination was fair, valid and based on the syllabus.” The reader can make of that what they will.

The Competition

Determine the errors, ambiguities and sillinesses in the 2017 WA Applications Exam, Part 1 and Part 2(Here, also, is the Summary Exam Report. Unfortunately, and ridiculously, the full report and the grading scheme are not made public, and so cannot be part of the competition.)

Post any identified issues in the comments below (anonymously, if you wish). You may post more than once, particularly on different questions, but please don’t edit on the run with post updates and comments to your own posts. You may (politely) comment on and seek to clarify others’ comments.

This post will be updated below, as the issues (or lack thereof) with particular questions are sorted out.

The Rules

  • Entry is of course free (though you could always donate to Tenderfeet).
  • First prize, a signed copy of A Dingo Ate My Math Book, goes to the person who makes the most original and most valuable contributions.
  • Consolation prizes of Burkard’s QED will be awarded as deemed appropriate.
  • Rushed and self-appended contributions will be marked down!
  • This is obviously subjective as all Hell, and Marty’s decision will be final.
  • Charlie, Paul, Burkard, Anthony, Joseph, David and other fellow travellers are ineligible to enter.
  • Employees of SCSA are eligible to enter, since there’s no indication they have any chance of winning.
  • All correspondence will be entered into.

Good Luck!

Updates:

Well that worked well. Congratulations to Number 8, who wins by default. Details are here. We’ll attempt another competition, of hopefully broader interest, in the near future.

Numberphile and the Cult of Collegiality

Mathologer recently posted a long video addressing the “proof” by Numberphile of the “astounding result” that 1 + 2 + 3 + … = -1/12. As well as carefully explaining the underlying mathematical truth, Mathologer tore into Numberphile for their video. Mathologer’s video has been very popular (17K thumbs up), and very unpopular (1K thumbs down).

Many who objected to Mathologer’s video were Numberphile fans or semi-literate physicists who were incapable of contemplating the idea that Numberphile could have gotten it wrong. Many others, however, while begrudgingly accepting there were issues with the Numberphile video, strongly objected to the tone of Mathologer’s critique. And it’s true, Mathologer’s video might have been improved without the snarky jokes from that annoying cameraman. (Although, awarding Numberfile a score of -1/12 for their video is pretty funny.) But whining about Mathologer’s tone was mostly a cheap distraction from the main point. Fundamentally, the objections were to Mathologer’s engaging in strong and public criticism, to his lack of collegiality, and these objections were ridiculous. Mathologer had every right to hammer Numberphile hard.

Numberphile’s video is mathematical crap and it continues to do great damage. The video has been viewed over six million times, with the vast majority of viewers having absolutely no clue that they’ve been sold mathematical snake oil. Numberphile made a bad mistake in posting that video, and they’re making a much worse mistake in not admitting it, apologising for it and taking it down.

The underlying issue, a misguided concern for collegiality, extends far beyond one stupid video. There is so much godawful crap around and there are plenty of people who know it, but not nearly enough people willing to say it.

Which brings us to Australian mathematics education.

There is no shortage of people happy to acknowledge privately their frustration with or contempt for the Australian Curriculum, NAPLAN, VCE, AMSI, AAMT, MAV, teacher training, textbooks, and on and on. Rarely are these people willing to formally or publicly express any such opinions, even if they have a natural platform for doing so. Why?

Many feel that any objection is pointless, that there is no hope that they will be listened to. That may well be true, though it may also be self-fulfilling prophecy. If all those who were pissed off spoke up it would be pretty noisy and pretty difficult to ignore.

More than a few teachers have indicated to us that they are fearful of speaking out. They do not trust the VCAA, for example, to not be vindictive. To us, this seems far-fetched. The VCAA has always struck us as petty and inept and devoid of empathy and plain dumb, but not vengeful. The fear, however, is clearly genuine. Such fear is an argument, though not a clinching argument, for remaining silent.

It is also clear, however, that many teachers and academics believe that complaining, either formally or publicly, is simply not nice, not collegial. This is ridiculous. Collegiality is valuable, and it is obviously rude, pointless and damaging to nitpick over every minor disagreement. But collegiality should be a principle, not a fetish.

At a time when educational authorities and prominent “experts” are arrogantly and systemically screwing things up there is a professional obligation for those with a voice to use it. There is an obligation for professional organisations to encourage dissenting voices, and of course it is reprehensible for such organisations to attempt to diminish or outright censor such voices. (Yes, MAV, we’re talking about you, and not only you.)

If there is ever a time to be quietly respectful of educational authority, it is not now.

Eddie Versus the Forces of Woo

No one appears to have a bad word for Eddie Woo. And no, we’re not looking to thump Eddie here; the mathematics videos on Eddie’s WooTube channel are engaging and clear and correct, and his being honoured as Local Australian of the Year and as a Top Ten Teacher is really cool. We do, however, want to comment on Eddie’s celebrity status and what it means.

What do Eddie’s videos exhibit? Simply, Eddie is shown teaching. He is explaining mathematics on a plain old whiteboard, with no gizmos, no techno demos, no classroom flipping, rarely a calculator, none of the familiar crap. There’s nothing at all, except a class of engaged students learning from a knowledgeable and engaging teacher.

Eddie’s classroom is not the slightest bit revolutionary. Indeed, it’s best described as reactionary. Eddie is simply doing what good maths teachers do, and what the majority of maths teachers used to do before they were avalanched with woo, with garbage theories and technological snake oil.

Sure, Eddie tapes his lessons, but Eddie’s charmingly clunky videos are not in any way “changing the face of mathematics teaching“. Eddie’s videos are not examples of teaching, they are evidence of teaching. For actual instruction there are many better videos out there. More importantly, no video will ever compare to having a real-live Eddie to teach you.

There are many real-live Eddies out there, many teachers who know their maths and who are teaching it. And, there would be many, many more real-live Eddies if trainee teachers spent more time learning mathematics properly and much less time in the clutches of  Australia’s maths ed professors. That’s the real message of Eddie’s videos.

Chicken Shit

The ACCC has released guidance on the meaning of “free range eggs”, to come into force in April. There are a number of conditions for hens to be designated free range, but the clear mathematical requirement is that the chickens be subject to “a stocking density of 10,000 hens or less [sic] per hectare.” This compares to the maximum of 1500 hens per hectare recommended by the CSIRO. And by Choice. And by the Humane Society International. And by the RSPCA. And by pretty much everyone except Coles and other industry thugs.

The ACCC is just the messenger here, their guidance mirroring the Australian Consumer Law (Free Range Egg Labelling) Information Standard 2017, passed last April. The legislation was introduced by the Minister for Small Business, Michael McCormack. It was McCormack who took credit for the definition of stocking density:

 … my decision takes into consideration the views of consumers, advocacy groups and industry, and provides a sensible balance with a focus on informing consumers – so they can make the choice that’s right for their needs.

The reader can assess whether McCormack’s “consideration” has resulted in anything remotely resembling “sensible balance”, or in the ability of consumers to make an informed choice. Or, rather, whether Minister McCormack is simply another National Party asshole.

Downwardly Mobile

In response to France’s move to ban mobile phones from schools, now other countries are considering the same.

Well, sort of. Since 2010, France has already banned mobile phones from classrooms; what is controversial is the French proposal to ban mobiles from schools entirely. So, countries like England and Australia are only actively considering what France has accepted without question for years.

Of course, following the consideration to do the blindingly obvious, there is the backlash from the professionals. The ABC quotes NSW Secondary Principals’ Council president Chris Presland as saying

We talk about trying to stimulate STEM education in our schools … it seems quite bizarre that we’re talking about banning the most obvious forms of technology at our disposal. 

Dr Joanne Orlando, an “expert on children and technology” at UWS is also against any such ban. Responding to government comments, Dr. Orlando responds that

 it takes us a few years back from all the work we are doing in education and training … There are so many new ways that mobile devices can add to the classroom.

Thank God for experts.