Is Cognitive Science Full of Crap?

 

[Note 5/24/12: I’m away from the desk this week. This post originally ran on February 28, 2011, and stirred a lot of online discussion; still fully current, and I suspect will be so for a while.]

Is cognitive science full of crap? A biophyics researcher recently asked this of a cognitive science researcher. The latter answered with spirit. My own answer is that of course cog sci is full of crap — except when it’s not. Which makes it like most science, only more so.

It started when Cambridge University memory researcher Jon Simons posted a lament about how proposed UK science-funding cuts especially threaten young, developing researchers. The cog-sci debate broke out when University of College London biophysicist David Colquhoun suggested that perhaps precious funding might be better used if less were spent on cognitive science:

I couldn’t agree more about the very real danger posed to early-career and even mid-career scientists by the lack of smallish responsive mode grants.

But being in a different area, I may see the problem slightly differently. At the risk of being lynched, I’ll have to admit that I sometimes sigh when see the next “new phrenology” study come out. Only too often the results are uninterpretable (though university PR departments love the fact that, however trivial, they make headlines). The equipment is enormously expensive and perhaps some of that money could be better spent (for example, on fundamental biophysics!).

When pressed for examples, Colquhoun named a couple of studies, most prominently a 2000 study about the “taxi drivers’ hippocampus.” He was referring to work at the lab of Eleanor Maguire, who found that London taxi drivers, who to earn their licenses must pass a horrifically difficult navigational and geographic exam called The Knowledge, had bigger hippocampuses than most people did.

Hippocampuses play an vital role in memory and spatial navigation. As the paper noted, the bigger hippocampi in London cab drivers might mean one of (at least) two things: That memorizing the streets and routes of London made their hippocampuses grow; and/or that having big hippocampuses to start with made you better able to memorize enough routes and streets of the Knowledge to pass it. The paper leaned toward the former explanation.

Colquhoun, however, while admitting he didn’t know the study that well, said he was exasperated both with the sorts of big conclusions often produced by brain-imaging studies studies and with the hype they generate. Colquhoun was drawing on a strain of criticism that had already pained Simons and many of his readers. The charge of “new phrenology” carries a special edge, since phrenology, a sort of skull-reading 150 years ago, was an ugly hoax that now looks one step this side of voodoo.

Even if this argument fades at Simon’s blog (Simons now has a newborn at home, which might slow things down a bit), it will have legs elsewhere. And well it should. It speaks to two important issues: The difficulty of doing a difficult science well, and the difficulty of writing about it well.

As it happens, I can speak a bit to the taxi-driver example, for a while back back, puzzled over a spectacular spatial memory lapse of my own, I spent a lot of time researching spatial memory and the hippocampus. I also write often about cognitive and behavioral science, and am working on a book right now about behavioral science. And I think Colquhoun has a point. The behavioral sciences, faced with figuring out excruciatingly difficult problems, have often come up with some wacky explanations.

Yet I think Colquhoun overplays his point — and that he chose a poor example. For the taxi-driver study shows not the wacky side but a strong side of recent cog-sci research.

The taxi-driver study got splash precisely because it suggested and framed an interesting hypothesis — perhaps hard memory work grows your hippocampus — in a way that even casual lay readers could understand. And as it happened, the science has held up.

If that 2000 study was the only study, then it wouldn’t mean much. As I noted above, its key finding — the big hippocampi in taxi drivers — might have been happenstance, or people with big hippocampi might have been either attracted to taxi driving or had better memories to start with and so could better pass the rigorous navigation test. At the time, Maguire was pushing the edge a bit proposing that the hippocampi had in fact grown.

To her credit, however, Maguire spent much of the following ten years running a bunch of other studies to sieve out those and other confounds. She even did some before-and-after studies that found that taxi drivers’ hippocampi grew as they prepped for the test. Taken together, it’s a pretty striking body of work that supports her essential claim well. It also supports and gets support from a huge body of other work on memory and the hippocampus. People still debate the hippocampus’s core, elemental function. But the taxi driver studies happen to be of real consequence and rigor, part of a thoughtful, thorough, ongoing line of research that ties into other lines. And this time, the simple clean story (e.g., memory demands can grow your hippocampi) seems to be proving scientifically rigorous and relevant.

The very best imaging research does this, as does work in cog sci drawing on other methods. Helen Mayberg’s, for instance, is another remarkable line of work that leans heavily on imaging but ties into structural, autopsy, and genetic work to produce real value.

Colquhoun is right in saying there are a lot of crap imaging studies — enough, in fact, that one’s first reaction probably should be, “Interesting if true,” partly because the press often hypes up the most simplistic ones. This happens when a field races ahead. But sometimes racing ahead is how you distinguish the squishy ground from the solid.

I don’t know enough about David Colquhoun’s discipline to make a call on whether biophysics deserves funding more than cognitive science. I’d guess that his field has crap studies that get funded and good studies that do not. As to his frustration about hype: I can understand it, and I’d love to see more good ideas on how to check it. But Colquhoun should probably accept the outsized attention to cognitive science as inevitable. Cognitive and behavioral sciences will always get more play, because people are damned interested — as they must be — in what makes other people tick.

To expect otherwise — to expect most people to pay more attention to, say, particle physics or ion-channel dynamics than psychology and cog sci — is to hold expectations defying the evidence.

Mentioned:

Maguire, E. A., Gadian, D. G., Johnsrude, I. S., Good, C. D., Ashburner, J., Frackowiak, R. S., Frith, C. D. (2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc Natl Acad Sci U S A 97(8), 4398-4403 doi:10.1073/pnas.070039597.

PS: I managed to misspell Colquhoun’s last name 11 times in the original. Just fixed them. Apologies to DQ. Mar 10, 2011, 12:49 pm GMT