We’ve never paid particular attention to Scott Alexander (pseudonym) and his blog, Slate Star Codex (now Astral Codex Ten). Long ago, we ran into a few SSC posts and our very vague memory is that we liked them. Then, while reading more on the New York Times‘s crucifying of Donald McNeil, we came across the Times‘s smelly article on SSC, including the gratuitous and sleazy outing of Alexander.* We explored a little, including reading this thoughtful critique of SSC. Still, we didn’t, and don’t, look to have an opinion on SSC, or on the “rationalism” in which it is supposedly immersed. (We did however, come to the opinion, that the New York Times has lost its collective fucking mind.)
A couple days ago, however, and completely by chance, we ran into an old, 2014 post on SSC. Alexander is a psychiatrist, and this post was on his long and futile attempt to conduct a study of a screening test for bipolar disorder, a test that Alexander suspected was of limited worth and knew was being widely misused. From the introduction to Alexander’s saga:
You ask patients a bunch of things like “Do you ever feel really happy, then really sad?”. If they say ‘yes’ to enough of these questions, you start to worry.
Some psychiatrists love this test. I hate it. Patients will say “Yes, that absolutely describes me!” and someone will diagnose them with bipolar disorder. Then if you ask what they meant, they’d say something like “Once my local football team made it to the Super Bowl and I was really happy, but then they lost and I was really sad.” I don’t even want to tell you how many people get diagnosed bipolar because of stuff like this.
Alexander’s post struck a particularly strong chord. Our father was a clinical psychologist, and he saw the worst. The whole Cuckoo’s Nest thing. More importantly, he didn’t think the not-worst was generally all that much better. In general, he regarded his psychologist and, especially, psychiatrist colleagues as arrogant, narrow-minded, drug-pushing quacks. Alexander’s story is from a different angle, but it fits right in.
Alexander’s post is alternately hilarious and horrifying. Read it. And then scream.
*) The kind of thing that a toady like James Massola would do.
This post was labelled “not mathematics, psychology”. Fair enough. However, there is a lot of mathematics in psychology.
Let me give you one of my favourite examples. I have read in a Year 12 psychology textbook that a random sample is a sample chosen from a population in which every member of the population has the same chance of being selected.
I offer the following example for your consideration.
We have a population of 10 people, 5 of whom are men and 5 of whom are women. I want to choose a random sample of 5 from this population. So I flip a coin. If Heads, I choose all the men. If Tails, I choose all the women.
Cute. Although I think the cuckoo’s nest guys had much more pressing problems than bad definitions.
Counterpoint: that *is* a random sample, it’s just not a *representative* sample either way. That’s basically systematic sampling in the event that the people are lined up alternating in gender.
However, that statement in the textbook is still false. Rolling a die and picking all the men if it lands on 1 and all the women if it lands on anything else is *also* a random sample despite the men being much more likely to be picked; it’s just *biased*.
I beg to differ.
A simple random sample is a way of realising the assumption in statistics that random variables are “independent and identically distributed”. This assumption is important is all elementary statistics.
The problem with “Heads -> choose the men” is that the results are not independent of each other. If one man is chosen, they are all chosen.
The definition in the psychology text book does not mention independence, and therein lies its fault.
Three tangential points.
1. The same problem arose in conscription during the Vietnam War. Young men were chosen on the basis of their birthday and hence the sample of conscripts was not random. If two men had their birthday on the same day then the results of the ballot were the same for them.
2. Independence is a subtle concept in probability.
3. Statistical methods are based on random samples, not representative samples. Whether a sample is representative or not does not come into the statistical method. This surprises many people. Random samples are likely to be representative, but in some cases they won’t be. There are ways of creating random samples that are more representative (e.g. stratification). But the mathematics underpinning elementary statistical methods is based on the sample being random.
Yeah, I remember that fight. The huge problem people had with conscripting kids to go kill and die in Vietnam was that the selection process wasn’t truly random.