The breadth of skepticism

EDIT: Note – I do realize some of my own views may be informed by my preconceptions and preferences. I’m not immune to wishful thinking or just being plain wrong, by any means.

A little while ago, amidst the clamour of Elevatorgate (can we stop appending -gate to drama/scandals?), there was an issue raised at TAM 9. There was basically a bit of argument on the Diversity in Skepticism panel over just what we should apply skepticism to.

Now, having been in the skepticism/atheism movement for a couple years, I’ve definitely noticed a big split, which is caused almost entirely by politics. On one side you have very left-wing pro-science, atheistic people; on the other side you have centre-right pro-science atheistic people. Obviously the lines aren’t quite as clear as all that, but in the absence of arguments over the truth of religious or paranormal claims, most pub arguments between skeptics come down to politics. If you wanted to phrase it in terms of popular academic white dudes, you could see it as a bit of a PZ Myers / Sam Harris split; I think a better description is that it’s a split between people who acknowledge privilege and those who don’t.

Since I’m definitely not getting sucked into an argument about privilege here, please read about Your Invisible Knapsack before saying anything. Actually, fuck it, read this too.

Now, at that TAM9 panel, DJ Grothe said he gets letters from conservative white dudes, who feel excluded from meetings. In the interest of being constructive I’ll refrain from speaking my mind on that subject, but later Jamila Bey suggested we ought to confront things like poverty, police violence, and the drug war. DJ Grothe responded by saying we ought to be attempting to preserve a diversity of views, rather than adopting any political stance. I’ve seen similar arguments from people who consider themselves skeptics but not atheists, and who dislike the association of skepticism with atheism.

It seems to me, that if we are going to claim some kind of intellectual honesty, we cannot apply the tools of skepticism selectively. If we can apply critical thinking, self-reflection, logic, rationality, and science to obvious nonsense like homeopathy, the anti-vaccination movement, or reiki, why can’t we similarly apply them to religion? And similarly, why can’t we apply them to politics?

Many, if not most, political questions can be approached scientifically, or at least critically. If someone claims a policy will have a certain effect, check if similar policies have had similar effects in similar contexts. Does a law do what lawmakers claim it does? Track its effects over time and validate it. Do mandatory minimum sentences in drug cases reduce the number of dealers? What are the broader effects? Do some laws disproportionately affect certain groups? Why? Are the laws being applied consistently, or are their enforcers exercising some bias? Does cutting government spending during a recession improve the condition of a nation and its citizens? Does it prolong the recession? Does restricting abortion reduce the number of abortions? Does it raise the number of women injured in amateur abortions? Does abstinence-only sexual education reduce the frequency of sexual activity among minors? Does it increase the age at which they start? Does it reduce teen pregnancies and the incidence of STIs?

You may note that many of these questions, far from being independent of scientific or religious claim, are intimately bound up in them. Public health work is hugely central to many government policies, but to an outsider it seems to play a relatively small role in informing policy decisions. Why is abortion such a big issue? Largely because religious groups claim fetuses, embryos, and earlier forms are sentient individuals. Or rather, that they’re imbued with a soul, often from the point of conception. Racial studies show that black youths are disproportionately arrested for drug offences, or stopped and searched by police, despite not having particularly different incidences of use than white youths. Shouldn’t this inform law enforcement policy? A casual look at the numbers shows American Muslims are overwhelmingly responsible for reporting suspicious or extremist behaviour or rhetoric to the FBI. Shouldn’t this inform public policy, and ideally even public opinion?

Nothing should be off-limits for inquiry – indeed, active inquiry should be encouraged. This has caveats of course, that such inquiry should be informed, ethical, and checked independently. After all, one of the reasons homeopathy studies keep being funded is the claim that it’s understudied, or frequently used. But with limited resources, we shouldn’t be studying things which violate a broad range of physical laws. But we could, for example, study exactly how many people use it, why they use it, and whether they replace actual medical care with it.

I’ve noticed sometimes that the “nothing should be off-limits” line is sometimes used to justify racist or sexist or transphobic or homophobic rhetoric (although less of that latter one these days). This is, simply, bullshit. Bad research is never justified. I’ve seen people point to the racist mumblings of JP Rushton to justify their racist ideas, without any understanding that his work is essentially pseudo-science, and repudiated by pretty well everyone in evolutionary psychology (a field more vulnerable to nuts than most). You could also consider David Starkey, a historian who’s come up with some explanations for the riots in the UK which are both racist and incorrect. And I should point out that that last is a bit tautological – things cannot be both racist and correct, and if they are correct, they might not be racist. Though you’d want to be pretty fucking sure it was correct, and not just you assuming it was correct, or using your own preconceptions to inform your half-assed study. I’m looking at you, Satoshi Kanazawa.

So, essentially. It’s incredibly stupid to assume that we can be skeptics while refusing to apply skepticism to our own social and political beliefs. I’m not saying we’ll find some universally correct position, the way Sam Harris seems to think, but we can do a hell of a lot better than we do now.

Can people get better?

I attended a talk last night co-hosted by McGill Freethought, CFI Montreal, and the McGill OSS, on science and the paranormal, hosted by Dr. Joe Schwarcz.

It was pretty good, though the material was familiar. He went through several problems that cause people to have mistaken beliefs, like coming to the wrong conclusion from accurate observations, being poor observers (like everyone is), coming to unwarranted conclusions from insufficient information, etc. It’s part of skepticism I like, because in principle it’s quite egalitarian. While it annoys me, the charge that atheists are elitist isn’t always misdirected. I adore Daniel Dennett, and understand what the people who founded it were attempting, but the Brights movement bugs the hell out of me.

So far so good. He did some fun demonstrations of stage magic to show how we can be mistaken, talked about some issues like UFOs and psychic surgery, and was generally entertaining.

But I think there were some things that really showed the gap between the younger group of skeptics coming up now, and the old-school people in the movement. Dr. Schwarcz would casually talk about Randi, Sylvia Browne, Uri Geller, the million dollars, the Carson Show, that the Reagans relied on astrology… All of it stuff that was very relevant to the skeptical movement in the 70s-80s, and maybe even into the 90s, but most younger people in the skeptical movement haven’t even heard of some of these people. It’s not relevant. Hell, I’m 27 and some of it’s not relevant to me. I know who James Randi is, sure, but that’s not true for everyone.

And that doesn’t even count that there might have been people who were just curious who’d come to the talk. How are they supposed to know who these people and things are, especially if they’re under 30 or so?

I know this is reflected by Dr. Schwarcz not being a spring chicken himself, and from what I know he’s very much a part of the old guard types who were involved in CSI-COP and similar groups. And I don’t mean to rag on the elders of our movement – we wouldn’t be where we are without them. But a movement that wants to be relevant needs to work at it, and to keep moving along.

The other thing that bugged me was the overall pessimism. Several of us, almost exclusively younger people, asked what we can do to help people understand the cognitive problems that lead to supernatural beliefs, to help family or friends who are hooked into astrology, or mediums, or ‘healers’, or other such things. And we were told that there’s pretty much nothing we can do.

Now, I get that it’s hard. Most of these beliefs are self-reinforcing, and getting anyone to change their beliefs about anything is hard at the best of times. And there needs to be some desire to change them, which many people just don’t have. But to act like it’s fruitless is again elitist, and defeatist. What’s the point of organizations like the three that hosted this event, if there’s no way to educate people?

Too, very few skeptics have always been skeptical or critical thinkers. None of us are for everything, of course. Everyone has blind spots, things they’re unable to examine rigorously, and all we can do is try to minimize those. I was religious, once upon a time. I believed in all sorts of things that I hadn’t even considered critically. I even though there might be something to homeopathy – I’d vaguely heard about Benveniste’s experiments, didn’t know much about homeopathy itself, figured it was a vaccination type thing… was I a write-off for a skeptical or atheist movement? No.

And I know lots of people like myself who are involved in skeptical or atheist groups. People who were devout religious believers, who were into ghosts or alien visitations, energy healing, the works. Something happened to all of us. Maybe we had something bad happened and re-examined our faith. Maybe a prediction failed to come true, or a relative who went for reiki died of their actual illness. Maybe we had a chiropractor offer to cure a genetic disorder.

And maybe it wasn’t something bad. For myself, I stuck around in science, and a lot of my faith faded as I realized I didn’t need it. I started reading some blogs, and my girlfriend introduced me to Daniel Dennett, whom she’d been reading for her art and AI class. I was introduced to the skeptical movement through science, largely scienceblogs.com, and I learned there and in my schooling how to examine people’s claims more critically.

So it’s worth it. Lots of people will ignore us, sure. We’ll be called arrogant and elitist, we’ll be told we’re close-minded or ignorant, kill-joys or meanies. And we can have a discussion within our movement on how to present ourselves, how to reach out, how to respond.

But we can’t just turn inwards.

The Lorne Trottier Public Science Symposium on Pseudoscience

So, Monday and Tuesday were the Lorne Trottier Public Science Symposium, a yearly event at McGill which provides a free public forum for people to watch scientists talk about science. Basically.

I’ve not gone before, but this year it was on pseudoscience, so how could I resist? The Friday event I mentioned last post kicked it off.

Monday afternoon there was a ‘press conference’ at the McGill Faculty club (which is posh as hell), which was really more of a student Q&A with the speakers: Michael Shermer, David Gorski (of Science-based medicine and the blog of a friend of his), and Ben Goldacre of Bad Science.

It was pretty good. There were questions about what ought to be covered by health insurance, the placebo effect, the infiltration of quackery into academics, how to engage with people, and so forth. It was a nicely intimate setting.

After that myself and a friend met up with some folks from the CFI to talk about events before heading into the Monday night talk.

The event was opened by the Dean of Science at McGill and the Provost, with the introduction by Joe Schwarcz of the OSS.

Shermer was up first. I don’t know him as well as the others, and I’ve never read any of his writing besides a blog post or two. He was very brash and pugnacious, and did a good talk on how cognitive issues in humans make it difficult for us to think and act skeptically. Useful talk, though up here he comes across as very American.

Gorski was after him, and he gave a talk focused on breast cancer quackery, and gave some good advice on how to deconstruct cancer testimonials. He’s definitely as big a nerd as the internet indicates – while I enjoyed his talk, I think he’s still getting used to addressing large public audiences instead of medical-scientific groups or individual patients.

Goldacre was probably the best speaker that night, though since I have the usual Canadian love for an English accent I might be biased. He did a brisk talk on supplements and quacks, and some of the legal trouble he’s had.

Shermer was the only one who managed to keep it pretty light. People dying of cancer is rarely funny, so Gorski’s speech ended up being a bit of a downer, and Goldacre ended up talking about Matthias Rath, who’s basically one of the more evil people walking the earth today. And the only reason I can write that is because I’m in Canada and not the UK.

There was a question period too! Sadly, the only questions I remember were from nuts. The first was a self-proclaimed lawyer who said that he’d examined each and every alternative and mainstream medical treatment, and knew that all alt-med was bunk. Except, of course, for Rrrroyal Rrraymond Rrrrife, whose experiments in the 1930s were never replicated! The frequencies were lost sirs, and why has the scientific community never embraced this great man’s work I ask you yes you sir what are you afraid of, that you will not reconstruct the great work of Rrrroyal Rrrraymond Rrrrife? Even Gorski ended up butting in to ask if he was going to ask an actual question, which he spent 10 minutes doing. Given Gorski’s a cancer surgeon/researcher and had spoken on the topic I was impressed by his restraint.

The other one was told he could ask a one-sentence question, and bargained for 30 seconds. He then proceeded to ask Shermer at great length about the Kennedy assassination (!), for 39 seconds. I was impressed, both because he’s clearly been on about this for 40-some odd years at least, and that he managed to pack a question into 39 seconds. If only the first nut had been so brief…

I hung around after and got to meet Ben Goldacre, who’s a great bloke. Very friendly, easy-going, and pleased to sign something for me. And I got to meet David Gorski! You’ll have to forgive the fannish-ness, but given the huge influence his writing has had on me in the past few years I was pretty excited.

A bunch of us CFI and Freethought types went out after for a few drinks after, and had some good conversation.

Tuesday James Randi gave a presentation at McGill. Though I’ve not read any of his books, I knew who James Randi is, and he’s a living legend. Much like Harry Houdini, he started out as a stage magician and ended up as a skeptic and investigator, even working with scientific committees to critically examine claims which scientists are not always equipped to investigate.

The talk was excellent. Randi is tiny, but he has an amazing stage presence. Lots of old-but-good jokes, some conjuring tricks, and an explanation of why he’s so devoted to skepticism. Long story short – charlatans and frauds hurt people by preying on their belief. He showed how he exposed Peter Popoff, how he demonstrated psychic surgery, mentalism, etc. I was impressed. Magic fascinates me, though I’ve never really been tempted to take it up, and I always love to find out how the trick was done.

I’d talk more about Randi, but as I don’t know him as well as the Monday speakers, I don’t have as much to say.

If you’d like to see or listen to the talks, you can go to the following link:

http://bcooltv.mcgill.ca/ListRecordings.aspx?CourseID=3113

Pseudoscience and The Living Matrix – not even wrong.

So McGill hosts these “Freaky Friday” events, in which McGill scientists are supposed to explain some of the actual science behind films and pop culture. Ostensibly, anyways. Mostly it seems to be a science lecture followed by a movie. You can find details on them here.

I was put off of them last year by the UFO ones, in which a professor abused the philosophy of science to tell us that we’ve totally been visited by UFOs and there’s a massive global cover-up. That wasn’t so cool, especially since there’s enough disinformation out there already. We don’t need people muddying the waters even more.

But I’ve gone to two this year, and they’ve both been pretty cool. The last one was by a biogeologist talking about the work being done looking at indirect evidence of life earlier than was previously thought, using Carbon-12/13 ratios and the atmospheric ratios of different Sulpher isotopes. It was neat. I never thought finding old rocks, grinding them up, and then analyzing them would be so cool, but it was. The movie for that was Night of the Triffids, which was an awesomely bad old sci-fi flick.

Tonight was a lot closer to my heard. Joseph Schwarcz was the presenter, which was cool. The talk was on the growth of quackery, and as he made clear it really hasn’t changed much – more sophisticated maybe, but the claims and the content are pretty similar.

He brought us through snake oil salesmen and patent medicine, talked about Houdini’s skepticism, and showed some more modern quackery like Asea http://www.teamasea.com/, which is a 5% salt water solution. And really, it doesn’t even scratch the surface. I’m not going to go into the history of quackery, or all the various forms and authors, but suffice to say it was interesting. Check out Dudley J. Leblanc and Hadacol for an example from the 50s.

I’ll just mention one other, since it’s so overblown. Bill Nelson (you have to read his ‘bio’) invented the EPFX-SCIO, which claims the following:

The EPFX-SCIO scans the body for 9000 frequencies, each associated with a different compound, much as anti-virus software would do for a computer. The EPFX-SCIO operates at biological speeds (up to 1/1000 of a second) charting the resonance or responsse of the body to these frequencies, comparing them to a norm and ranking them in degree of reactivity, identifying both acute and chronic imbalances. Clients can then be provided information about the results and energetic therapy can be given to attempt a balancing or harmonizing of any aberrant frequencies. Offering over 200 biofeedback therapies in 72 modalities, it is the largest healthcare software package in the world, combining both eastern and western philosophies and techniques.

I don’t even know where to start…

After that we watched a film called “The Living Matrix”.

Let’s play a game. Watch the movie, and take a drink every time someone makes a claim that runs counter to basic science, or makes a factual scientific error.

Now that you’re out of the hospital, I’ll remind you that you can’t sue me for your alcohol poisoning.

The film is a painful mishmash of healing touch, energy medicine, ‘The Secret’-esque intentionality stuff, and a hefty dose of what I’ve started thinking of as “Fuckin’ magnets, how do they work” syndrome. I can’t even address their claims, since they often conflict. Near the beginning they claim that consciousness is not in the brain, and that the body does not need some sort of central organizer. But then in the second half, they spend a bunch of time wondering how the body can function without a central organizer.

They even wonder how cells can communicate with each other, and I foolishly thought they would start speaking about inter-cellular communication. Hah! No, they claim that some central organizer needs to send out simultaneous signals to all the cells at once (since they claim the different speeds of nerve transmission make complex behaviour impossible). They claim that the actual central organizer is the heart!

That’s right, we’re right back to fucking Aristotle. They say it can “imprint information” on the body using “sound waves, electricity, magnetism, and electromagnetism” and that the nerve tissue on the heart (which, you know, the brain uses to maintain a heartbeat) is actually to tell the brain what to do. Seriously – some of the same arguments as 2300 years ago when Aristotle argued the heart was the centre of consciousness. And how does it talk to cells? Well, they’ve got these receptors embedded in the cell membrane…

*facepalm* Which are used for chemical signalling between cells. Apparently they missed that bit in high school or first year bio. And none of this is about the more egregious nonsense about “fields” which is addressed a bit here.

Anyways. Suffice to say that me, the developmental biology student next to me, the one across the room, and the engineer sitting below me were all choking back astonishment and outrage for most of the film. The phrase “not even wrong” comes to mind.

The after talk was… interesting. There were a couple people who were really into it, and they talked about how they believed in intuition, or telepathy because they would think of someone and then that person would call them. This old nonsense always ignores the huge numbers of times when you’re thinking of someone and they don’t call.

So, it was fairly fun even if the film was frustrating as hell. Anyone care to explain how Rupert Sheldrake has a position at Cambridge? Oh, the they mentioned Adam Dreamhealer. Oh yeah.

You can see more thorough review of the film here.

I wonder sometimes if I’m even speaking the same language as these people.

The Universe does not conform to our desires.

So not much today. I went a pretty interesting lecture on music and pleasure, focusing on the dopamine reward system in the brain, especially the dorsal and ventral striatum. Anyone interested should check out one of the papers discussed there, which can be found here.

It was a good talk, and there were some interesting questions about combining PET and fMRI, and some speculation about physiological effects of music. Plus Brenda Milner was there, which is always like seeing a living legend.

While I try to think of a hypothesis that can be answered using a combination of unconnected fMRI studies, here’s a video of one of my favourite segments from Carl Sagan’s Cosmos.

Atheism

I might have alluded to this in previous posts, but I should address it firmly. I am an atheist.

It doesn’t seem like a particularly courageous declaration in my time and place (2010, Montreal). There are other countries, or other regions, where it would cause me trouble. Most certainly there are other times; the theocratic empire of Theohoplon, circa 2245-2378 would be a very bad place indeed.

But even given today’s declining numbers of self-identified religious believers, the word ‘atheist’ still carries connotations of willful denial, anger, and nihilism. When telling a non-believing co-worker on evening that I was going to a meeting about how to live a fulfilling life without God, she told me that it sounded ‘angry’. When I mentioned to a professor that I help run a club devoted to ‘skepticism, rationalism, and free inquiry’ she said it sounded ‘scary’.

So. Where do I stand vis-a-vis God/gods? In general, I don’t think about it much. On a daily basis, sure, but that’s mostly because I’m active. But it’s not as though I notice my lack of prayer before meals, or when friends are hopeful for a break. I think, if anything, it makes me a little embarrassed when swearing, because there aren’t many useful curses that are both non-religious and non-obscene.

God, so far as I can tell, doesn’t exist. It’s not provable, but it seems unlikely. I know this could theoretically make me an agnostic, but I prefer the term ‘atheist’ because it’s less ambiguous. Even if I have to preface arguments by stating I can’t absolutely reject the possible existence of some deity-like thing, I feel it’s worth it to have people know where I stand.

My atheist is tied in strongly with my rationalism (in the colloquial sense, not the strict philosophical sense). Thus, I tend to look askance as atheists who are also supernaturalists, with beliefs in ghosts and Atlantis and magic and alien abductions (though I admit the probable existence of aliens, there are good reasons to believe they’re a bit too far away to pop over for a cup of sugar and some livestock). This is another reason to call myself an atheist – agnosticism has a way of being associated with a certain fuzziness towards all sorts of daft notions, and so calling myself an atheist means I’m less likely to be pestered about guardian angels or the healing powers of crystals.

While this isn’t intended as a comprehensive look at atheism, here are a couple arguments I’ve found pretty persuasive on my side.

One is Epicurus’ paradox of evil, as restated by David Hume:

“Is he [God] willing to prevent evil, but not able? Then is he impotent. Is he able, but not willing? Then is he malevolent. Is he both able and willing? Whence then is evil?”

This is a question of theodicy, the question of why evil exists. There is a solution, as noted by Epicurus himself. God might be distant and uncaring, or actively malevolent. There are a number of theological solutions to the question (free will, mysterious ways, etc) but they have all been pretty unsatisfactory to me.

Another is an argument about morality, put forth my Plato. I’m divorcing it a bit from the context, because it’s the statement I’m concerned about. “”Is the pious loved by the gods because it is pious? Or is it pious because it is loved by the gods?”

Replace ‘pious’ with ‘good’. Thus, if a thing is good and therefore loved by the gods, it would be good regardless. And if it were good only because the gods loved it, then it’s goodness is fairly shallow, determined only by the liking of a powerful figure. If, say, patricide were right because Caesar willed it so, would that make patricide alright?

Both of the above are more arguments about specific things – they don’t really address God’s existence itself. For me, I find the most persuasive argument is this: everything that happens, can happen without God. God, therefore, is unnecessary. In the interest of parsimony and getting on with things without having to worry a lot about what He hypothetically thinks, let’s assume He’s not in the picture.

There’s a bunch more to it, of course. People have been arguing over these things for millennia, and a fair body of work has built up. But that’s some of it, above. Maybe not the core, even, but some important bits.

On egregious nonsense

So. Of all the pseudoscientific nonsense out there (and there’s a LOT), homeopathy bothers me more than any other I can think of offhand. I’m not entirely sure why, but I think it’s because homeopathy lacks even the prima facie sort of plausibility possessed by modalities like acupuncture or chiropractic.

The background for homeopathy has been covered much more expertly elsewhere, so I’ll just give a quick summary. Back in the 1800s, a doctor named Samuel Hahnemann took a bit of quinine (used as treatment for malaria). It’s also poisonous, and when he took it he felt it induced symptoms similar to those of malaria. He decided that things which cure someone who is sick will create the symptoms of that illness in a healthy individual. This is the ‘like cures like’ concept of homeopathy, that by giving substances to healthy persons and noting how they feel, you can use those substances to cure those symptoms in a sick individual.

Note that this isn’t like vaccination – you aren’t acclimatizing their system or provoking an immune response so your body will recognize a virus later. It’s literally supposed to be curative.

There’s another principle. Hahnemann noticed that when he gave substances to his patients (like arsenic, for example) they ended up healthier when the substance was more dilute. This makes some sense – after all, you’ll be much healthier if you consume almost pure water or a sugar pill than if you consume arsenic or quinine (unless in the latter case you have malaria). Thus the second principle – dilution. Homeopathic ‘medicines’ are typically diluted to 20-30C, each C standing for a 1-in-10 dilution. So take 1mL of a 10mL solution, mix it with 9mL of pure water, and repeat 19-29 more times.

Now, a dilution of 10-23 or so will have about one molecule of the original substance in one mole of the solution (so 1 molecule per 16 grams) of water. It gets much less likely that you’ll find a single molecule of the original substance for every ‘to the minus’ that you go. So at 10-30 you’d have to drink oceans and oceans of a homeopathic solution to actually consume a single molecule of whatever substance was originally there.

Thus. We have a medical principle (like cures like) that’s not actually true, and a technical principle (of increasing potency with increasing dilution) that doesn’t actually make sense.

The key part I’ve left out? Succussion. Hahnemann said it would only work if you shook/tapped the substance between dilutions. He recommended a certain number of taps on the cover of a bible, to ‘potentize’ the substance. It won’t work otherwise, you see.

Homeopathy isn’t naturopathy (which I have my own problems with). It isn’t herbalism. It’s nonsensical magic that ought to be transparently useless at first glance.

I’ll leave you with the video below, and if you want more information I suggest the ‘Homeopathy’ category at Science-Based Medicine.

Multiple comparisons and coincidence

Statistics refers to various methods of looking at subsets of whole populations, called samples. Because it is usually impossible to use the entire population of whatever you are looking at (ie, when polling you can’t called /every/ American), we use samples of those populations and then extrapolate our results from that sample.

In science as in polling, you do want to try to have a representative sample, or to be comparing similar groups, so that your sample resembles the population. Single traits in the population, like height, tend to be distributed along a bell curve, as below – there are a lot of people of middle height, with some very short and some very tall people.

That’s a ‘normal’ curve, where the mean (average) equals the mode (most frequent score). There’s some fancier stuff about how tall or spread-out the curve is, but that’s essentially it.

Now, most statistical tests that you do in science assume that whatever you’re measuring is normally distributed (as in that curve above). They then compare the means of the difference groups in the experiment. See, there’s no way of knowing just from looking at two groups if they’re really that different, or if those differences are due only to chance. If you’ve measured, say, the reaction times of a medicated vs. a non-medicated group, you can’t just say ‘The medicated group had a mean reaction time of 500ms, and the non-medicated group had a mean reaction time of 450ms, therefore the medication slows reaction time.’ This might not be true – because of all the potential sources of error, the differences in reaction time can’t be solely attributed to the medication. What you need to know is /how/ different the reaction times are, and if you can say those differences are /significant/.

In the most general terms, what you would do here is find the difference between the two group averages, and then divide that by the total error measure of both groups. That’s a bit tricky to get into, so please for now take my word for it that you can get a mathematical representation of the error in the measurement (it’s like the +/- 0.05mL from high school science).

If you’re doing this by hand, you then take your score and compare it to a table of scores, based on your sample size. Normally, what we use as the significance cut-off is a p-value of < 0.05. What this means if our score falls into the top or bottom 2.5% on the curve above, and any difference we found between our groups is thus pretty unlikely to have occurred by chance alone. This is why polling data is reported as '19 times out of 20', since that's a 5% chance of the result being a coincidence.

Now, this is fine as it goes. However, if you want to compare more than two groups, things get tricky, because you'll often need to compare each group to one another, meaning instead of two comparisons you'll have 3 (3 groups), 5 (4 groups), 10 (5 groups), and so on (feel free to correct me if I messed up the comparison numbers there – the idea's still sounds).

Now, if I ask you to pull one gold coin out of a jar of 19 brass ones, you have a 5% chance of getting the right one. And the next time I ask you to do it, you still have a 5% chance, assuming you put the coin you pulled out first back inside. But – the second time you do it you have a 9.75% /cumulative/ chance of drawing the gold coin. The third time, you have a 14.3% chance. And so on. So how do we keep the chance 5% each time?

The same problem occurs during statistics. The more comparisons you do, the higher the likelihood of you getting a significant result through chance alone. Thus, we use a variety of techniques to make sure we're still looking at results which are unlikely to be found through chance alone. The method I use most often is a fairly conservative one (ie I'm more likely to discard a real difference as non-significant) called Bonferroni correction, which makes my significance criteria stricter for every comparison I do – essentially adding 20 coins for every time I plan to draw. Others do the equivalent of adding 20 coins after every new draw.

Now, where am I going with all this, other than giving some nifty insight into science and statistics?

Well, watch this and then come back:

From about 1:20 and again from 6:30 or so is what I'm thinking of. Coincidences seem marvellously unlikely to us, but the fact is that they are simply due to multiple comparisons. The more times we 'compare' things, the more likely we are to get a match, no matter how unlikely. The more people you meet, the more likely you are to meet someone else with a one-legged uncle names Charles. Spectacularly unlikely, yes, but if every person you meet counts as another comparison of means, or uncles, it becomes less and less unlikely.

This generalizes quite a bit. Lightning striking a church? There is a lot of lightning, and a lot of churches. Ignoring any real effects (churches are often on hills, have spires with metal bits, etc), by chance alone you'd almost have to /expect/ it to happen to some church over time.

Many things that seem unlikely are unlikely only in isolation – given the large number of things on Earth, and the fact that things happen over time, seemingly unlikely things are going to happen fairly regularly, especially when viewed in retrospect.

We need to try to correct for this. And that's what science strives for.

Tim Minchin Movie!

So, those of you in the skepticism community have likely heard of Tim Minchin, a comedian/musician. His piece ‘Storm’, a 9-minute beat poem, is being turned into a short animated film. Here’s the trailer.

For those of you unfamiliar with the original, here’s the poem.