Is Cognitive Science Full of Crap?

 

[Note 5/24/12: I’m away from the desk this week. This post originally ran on February 28, 2011, and stirred a lot of online discussion; still fully current, and I suspect will be so for a while.]

Is cognitive science full of crap? A biophyics researcher recently asked this of a cognitive science researcher. The latter answered with spirit. My own answer is that of course cog sci is full of crap — except when it’s not. Which makes it like most science, only more so.

It started when Cambridge University memory researcher Jon Simons posted a lament about how proposed UK science-funding cuts especially threaten young, developing researchers. The cog-sci debate broke out when University of College London biophysicist David Colquhoun suggested that perhaps precious funding might be better used if less were spent on cognitive science:

I couldn’t agree more about the very real danger posed to early-career and even mid-career scientists by the lack of smallish responsive mode grants.

But being in a different area, I may see the problem slightly differently. At the risk of being lynched, I’ll have to admit that I sometimes sigh when see the next “new phrenology” study come out. Only too often the results are uninterpretable (though university PR departments love the fact that, however trivial, they make headlines). The equipment is enormously expensive and perhaps some of that money could be better spent (for example, on fundamental biophysics!).

When pressed for examples, Colquhoun named a couple of studies, most prominently a 2000 study about the “taxi drivers’ hippocampus.” He was referring to work at the lab of Eleanor Maguire, who found that London taxi drivers, who to earn their licenses must pass a horrifically difficult navigational and geographic exam called The Knowledge, had bigger hippocampuses than most people did.

Hippocampuses play an vital role in memory and spatial navigation. As the paper noted, the bigger hippocampi in London cab drivers might mean one of (at least) two things: That memorizing the streets and routes of London made their hippocampuses grow; and/or that having big hippocampuses to start with made you better able to memorize enough routes and streets of the Knowledge to pass it. The paper leaned toward the former explanation.

Colquhoun, however, while admitting he didn’t know the study that well, said he was exasperated both with the sorts of big conclusions often produced by brain-imaging studies studies and with the hype they generate. Colquhoun was drawing on a strain of criticism that had already pained Simons and many of his readers. The charge of “new phrenology” carries a special edge, since phrenology, a sort of skull-reading 150 years ago, was an ugly hoax that now looks one step this side of voodoo.

Even if this argument fades at Simon’s blog (Simons now has a newborn at home, which might slow things down a bit), it will have legs elsewhere. And well it should. It speaks to two important issues: The difficulty of doing a difficult science well, and the difficulty of writing about it well.

As it happens, I can speak a bit to the taxi-driver example, for a while back back, puzzled over a spectacular spatial memory lapse of my own, I spent a lot of time researching spatial memory and the hippocampus. I also write often about cognitive and behavioral science, and am working on a book right now about behavioral science. And I think Colquhoun has a point. The behavioral sciences, faced with figuring out excruciatingly difficult problems, have often come up with some wacky explanations.

Yet I think Colquhoun overplays his point — and that he chose a poor example. For the taxi-driver study shows not the wacky side but a strong side of recent cog-sci research.

The taxi-driver study got splash precisely because it suggested and framed an interesting hypothesis — perhaps hard memory work grows your hippocampus — in a way that even casual lay readers could understand. And as it happened, the science has held up.

If that 2000 study was the only study, then it wouldn’t mean much. As I noted above, its key finding — the big hippocampi in taxi drivers — might have been happenstance, or people with big hippocampi might have been either attracted to taxi driving or had better memories to start with and so could better pass the rigorous navigation test. At the time, Maguire was pushing the edge a bit proposing that the hippocampi had in fact grown.

To her credit, however, Maguire spent much of the following ten years running a bunch of other studies to sieve out those and other confounds. She even did some before-and-after studies that found that taxi drivers’ hippocampi grew as they prepped for the test. Taken together, it’s a pretty striking body of work that supports her essential claim well. It also supports and gets support from a huge body of other work on memory and the hippocampus. People still debate the hippocampus’s core, elemental function. But the taxi driver studies happen to be of real consequence and rigor, part of a thoughtful, thorough, ongoing line of research that ties into other lines. And this time, the simple clean story (e.g., memory demands can grow your hippocampi) seems to be proving scientifically rigorous and relevant.

The very best imaging research does this, as does work in cog sci drawing on other methods. Helen Mayberg’s, for instance, is another remarkable line of work that leans heavily on imaging but ties into structural, autopsy, and genetic work to produce real value.

Colquhoun is right in saying there are a lot of crap imaging studies — enough, in fact, that one’s first reaction probably should be, “Interesting if true,” partly because the press often hypes up the most simplistic ones. This happens when a field races ahead. But sometimes racing ahead is how you distinguish the squishy ground from the solid.

I don’t know enough about David Colquhoun’s discipline to make a call on whether biophysics deserves funding more than cognitive science. I’d guess that his field has crap studies that get funded and good studies that do not. As to his frustration about hype: I can understand it, and I’d love to see more good ideas on how to check it. But Colquhoun should probably accept the outsized attention to cognitive science as inevitable. Cognitive and behavioral sciences will always get more play, because people are damned interested — as they must be — in what makes other people tick.

To expect otherwise — to expect most people to pay more attention to, say, particle physics or ion-channel dynamics than psychology and cog sci — is to hold expectations defying the evidence.

Mentioned:

Maguire, E. A., Gadian, D. G., Johnsrude, I. S., Good, C. D., Ashburner, J., Frackowiak, R. S., Frith, C. D. (2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc Natl Acad Sci U S A 97(8), 4398-4403 doi:10.1073/pnas.070039597.

PS: I managed to misspell Colquhoun’s last name 11 times in the original. Just fixed them. Apologies to DQ. Mar 10, 2011, 12:49 pm GMT

6 responses

  1. How many times have researchers said coffee is bad, then good then bad? Or wine. Or various other foods and chemicals and behaviors. One day we may actually know more but till then take everything with a grain of salt. Is salt good or bad these days?

    • What has that got to do with the article? And anyway, that is more to do with media representations of studies where they seem to scour journals, find one they can publicise and put it out there without context, and without explaining that these things are just correlational. It’s not the scientists themselves.

  2. From the abstract of the Taxi-driver article: “These data are in accordance with the idea that the posterior hippocampus stores a spatial representation of the environment and can expand regionally to accommodate elaboration of this representation in people with a high dependence on navigational skills.”

    This is classic cog-sci voodoo.  The data are also “in accordance” with the idea that the posterior hippocampus is where navigation Angels live, and can expand regionally to accommodate more Angels in people with a high dependence on navigational skills.  Less facetiously, the data are “in accordance” with lots of alternative explanations.

    The “hypothesis” that the hippocampus stores a spatial representation of the environment is pure, untestable dogma driven by the fundamental assumptions of the Cognitivist Ideology, which go back to 17th century myths about the mind as mirror, cognition as representation, etc. all of which have been radically undermined by both philosophers and scientists in the past 50 years or so.

    The problem is not that taxi-drivers have bigger hippocampi; it’s that the putative “explanation” of this is speculative non-science, as is so often the case with cogsci.  Taxi-drivers also do a lot of sitting in cars.  They also have to process a lot of visual information.  They spend a lot of time in the company of strangers.  Maybe they also drink a lot of coffee.  Etc. etc.  The number of possible explanations with which the data are “in accordance” is unlimited.

    That’s not to say that hippocampus size is unrelated to memory; it’s just that we don’t really know what memory is, and there is no good scientific reason to privilege the speculation that it involves storage of representations.  Braining imaging data may be interesting, but they don’t explain much of anything; using them to support cognitivist speculations is just bad science.  It’s like treating statistical correlation as a cause-effect relation.

  3. I echo Jon’s comment:. And the post above actually spells out how the original article was only the start of a long investigation by Maguire –and proposed not just voodoo, but a testable hypothesis (that memorization of London’s navigational routes can make the hippocampus grow), which Maguire then tested:

    To her credit, however, Maguire spent much of the following ten years running a bunch of other studies to sieve out those and other confounds. She even did some before-and-after studies that found that taxi drivers’ hippocampi grew as they prepped for the test. Taken together, it’s a pretty striking body of work that supports her essential claim well. It also supports and gets support from a huge body of other work on memory and the hippocampus. People still debate the hippocampus’s core, elemental function. But the taxi driver studies happen to be of real consequence and rigor, part of a thoughtful, thorough, ongoing line of research that ties into other lines. And this time, the simple clean story (e.g., memory demands can grow your hippocampi) seems to be proving scientifically rigorous and relevant.
    I’m all for jumping on crummy studies, and have stomped on quite a few myself. But we need be judicious and open-minded even as we apply our bullshit filters. 

  4. It strikes me that the target of Colquhoun’s criticism is better described as neuroscience than cognitive science.

    While neuroscience does play a role within the cognitive science framework, his criticisms seem aimed at solely at neuroimaging – specifically the kind of “neo-phrenology” that is rampant in neuroscience these days (hey, pretty coloured pictures sell…).

    There’s a whole lot of cog sci that doesn’t involve neuroimaging, so his complaints can’t be generalized to the whole field.

Leave a Reply

Your email address will not be published. Required fields are marked *