Social Psych’s Replication Problem Just Got Thornier

Much ado lately about the challenges of replicating key findings in social psychology. Almost everyone agrees it’s a good idea to test these key findings to see how they hold up; foundation stones should be solid.

Cambridge University social psychologist Simone Schnall, for instance, thought it was a good idea, and cooperated when a high-profile replication project attempted to see if they could replicate the findings in one of her papers. Then things got interesting. She wrote her experience up in detail at her department website.

Recently I was invited to be part of a “registered replication” project of my work. It was an interesting experience, which in part was described in an article in Science. Although the replicaton paper was published on 19 May 2014, my commentary describing concerns is still unavailable but it can be downloaded here.

Some people have asked me for further details. Here are my answers to specific questions.

Question 1: “Are you against replications?”

I am a firm believer in replication efforts and was flattered that my paper (Schnall, Benton & Harvey, 2008) was considered important enough to be included in the special issue on “Replications of Important Findings in Social Psychology.” I therefore gladly cooperated with the registered replication project on every possible level: First, following their request I promptly shared all my experimental materials with David Johnson, Felix Cheung and Brent Donnellan and gave detailed instructions on the experimental protocol. Second, I reviewed the replication proposal when requested to do so by special issue editor Daniel Lakens. Third, when the replication authors requested my SPSS files I sent them the following day. Fourth, I offered the replication authors to analyze their data within two week’s time when they told me about the failure to replicate my findings. This offer was declined because the manuscript had already been submitted. Fifth, when I discovered the ceiling effect in the replication data I shared this concern with the special issue editors, and offered to help the replication authors correct the paper before it goes into print. This offer was rejected, as was my request for a published commentary describing the ceiling effect.

I was told that the ceiling effect does not change the conclusion of the paper, namely that it was a failure to replicate my original findings. The special issue editors Lakens and Nosek suggested that if I had concerns about the replication, I should write a blog; there was no need to inform the journal’s readers about my additional analyses. Fortunately Editor-in-Chief Unkelbach overruled this decision and granted published commentaries to all original authors whose work was included for replication in the special issue.

Of course replications are much needed and as a field we need to make sure that our findings are reliable. But we need to keep in mind that there are human beings involved, which is what Danny Kahneman’s commentary emphasizes. Authors of the original work should be allowed to participate in the process of having their work replicated. For the Replication Special Issue this did not happen: Authors were asked to review the replication proposal (and this was called “pre-data peer review”), but were not allowed to review the full manuscripts with findings and conclusions. Further, there was no plan for published commentaries; they were only implemented after I appealed to the Editor-in-Chief.

Various errors in several of the replications (e.g., in the “Many Labs” paper) became only apparent once original authors were allowed to give feedback. Errors were uncovered even for successfully replicated findings. But since the findings from “Many Labs” were already heavily publicized several months before the paper went into print, the reputational damage for some people behind the findings already started well before they had any chance to review the findings. “Many Labs” covered studies on 15 different topics, but there was no independent peer review of the findings by experts in those topics.

For all the papers in the special issue the replication authors were allowed the “last word” in the form of a rejoinder to the commentaries; these rejoinders were also not peer-reviewed. Some errors identified by original authors were not appropriately addressed so they remain part of the published record.

Get the rest at Schnall’s post at the blog at the Cambridge University Department of Psychology.

Hat tip to the great Vaughan Bell channeling the magnificent Sarcastic_F. Founts of great leads the both of them.

Can a Movie About a Library Be Awesome?

Fixin’ to find out.

Cold Storage Teaser Trailer from metaLAB(at)Harvard on Vimeo.

Out at the Harvard Depository in Southborough, Massachusetts there are many stories to tell. How do the books come to and from campus nearly an hour away? What is the best way to store a library collection whose offsite holdings alone are mounting to ten million? What does it take to keep books at cold preserving temperatures and film reels at even colder ones?

Our upcoming documentary, Cold Storage, uncovers an ecosystem of laser scanners, UV fly zappers, cherry pickers and a mezzanine of machinery. It shows a place where books are sorted not by the methods of Dewey or those of the Library of Congress but by size.

In this trailer, take a peek inside the expansive interiors where our story begins and stay tuned for the debut of our film, which this summer will be incorporate the work of metaLAB alongside the projects of students from this past Spring’s Humanities Studio 1.

Coming soon is an experimental and interactive documentary to explore the HD as an offsite lens by which to examine the cultural and technical dimensions of libraries. (More: http://librarybeyondthebook.org/ )

Hat tip to Maryn McKenna channeling Alexis Madrigal.

Added a bit later:

¡Incredible! Sharp-eyed reader Liza Gross alerted me to Toute la mémoire du monde, (All the world’s memory), an eerily, beautifully anticipatory documentary by Alain Resnais about the Bibliothéque Nationale in Paris. Much of the filmic technique and many of the shots are like pre-echos of the Cold Storage film above. Lovely stuff proving that Yes, a movie about a library can be awesome.

The Week’s Brightest Glitter – My Favorite Reads

Mack McCormick, at home in Houston, December 2013.
Mack McCormick, at home in Houston, December 2013.

The Ballad of Geeshie and Elvie – NYTimes.com I can’t stop thinking about this article. Beautifully written deep-dive search for the identity of two iconic blues-women, the history of blues, the history of the history of blues, and the fuzzy line where digging for information becomes thievery. Hats off to John Jeremiah Sullivan, for enthralling writing, and to the New York Times Magazine design & tech teams for gorgeous multimedia that adds without distracting from Sullivan’s fine work. (Extra points that so much of this story rises from Houston, my hometown. Badly overlooked blues heritage there.)

Extraordinarily rich haul on mind and brain this week:

Love and Loss on the Seine Beautiful NatGeo story on the draw the Seine exerts on all sorts of people, including those in mental distress.

What Killed My Sister?, by Priscilla Long On schizophrenia; an explainer, and an homage to a sufferer.

My prism of postpartum depression Lovely account by T. DeLene Beeland.

Brain Stimulation Makes Man A Johnny Cash Fan? So it appears.

The cost of not caring: Nowhere to go. Much-needed look at our neglect of the mentally ill. At this point we’re chuckin’ ’em out to die. By Liz Szabo.

We cannot afford to study effect size in the lab. Honest hard look at a major problem in psychology studies in an age of big data. Working with people’s minds takes time.

Thousands of Toddlers Are Medicated for A.D.H.D. and Older Americans hooked on Rx: ‘I was a zombie’. More ugly from the medical system with the worst return-on-investment on earth.

Trigger alerts are dumbing down education. This is getting ridiculous.

Everything you know about teenage brains is bullshit Every headline written this way is overwrought and inaccurate. However, this article is a good solid calmer about fears that the intertoobz iz fizzing up yer teens’ brains.

Also, Is Low Blood Sugar Really Linked to Bad Behavior?, with the usual answer for such headlines.

But it seems true that Lonely people share too much on Facebook.

No, Timothy McVeigh Was Not Autistic The unifying feature of most mass murderers is rage, blasted outward at innocents by way of easily accessible firearms. No autism required.

 

On other fronts:

Arrested DevelopmentVirginia Hughes on four girls who apparently won’t age, and a 74-year-old scientist determined to figure out why.

Parasite Forces Host To Dig Its Own Grave Amazing grotesquery from Ed Yong.

Comb jellies — wee tiny sea critters — have some very strange brains. Carl Zimmer brings the goods.

Everything Science Knows About Hangovers—And How to Cure Them A sip from Adam Rogers’ fast-upcoming book, Proof: The Science of Booze.

Disrupting Laundry. On Silicon Valley’s laundry-app race The next new new thing in Silicon Valley. When are they going to clean that place up?

Two particularly devastating critiques of Nicholas Wade’s A Troublesome Inheritance: On the Origin of White Power | The Primate Diaries, Scientific American Blog Network, by Eric Johnson, and Nicholas Wade and race: building a scientific façade, by Jennifer Raff.

The best investment of the 20th century? Red wine. Damn. These things always seem so obvious after the fact.

Cinematographer Gordon WIllis passed away. Here’s some of his work.

In other filmmaking triumphs, A slug uses a running wheel

Showing Versus Telling: The Art of the Quick Sketch

Woolf Bell bio cover

Young writers are constantly told to show rather than tell. Here, in a sketch of Virginia Woolf’s father in Quentin Bell’s biography of Virginia Woolf, Bell shows how you can deliver the goods by telling quite a bit — and then, by showing just one or two things vividly, crystallize and embed all you have told.

Sir James Stephen, as he eventually became, was a courageous, intelligent and capable administrator. He was also, as he himself said, anything but a simple man. He inherited all the boldness of his father and his grandfather; he was formidable and implacable, driving others almost as hard as he drove himself, working long hours in the Colonial Office and making others work equally hard, still finding time for numerous contributions to the Edinburgh Review, dictating 3,000 words before breakfast. A monster of industry and learning, he was also a vulnerable and an unhappy man. His achievements fell far short of his ideals.… He was desperately shy, he was intensely pessimistic. He was so convinced of his own personal ugliness that he would not have a mirror in his room. He would shut his eyes rather than face an interlocutor. He wished he had been a clergyman, a recluse, anything but what he was. He was terrified of being comfortable and although he would not deny pleasure to others he was anxious to deny himself. Once he tasted a cigar and liked it so much that he resolved never to taste another. It occurred to him that he liked taking snuff — immediately he emptied his snuff box out of the window.

Virginia, of course, could write rather well too. Bouncing around her letters, I find one in which she writes a friend that, having finished The Waves and having been told by Leonard that the book is good, she is “as relieved as a girl with an engagement ring” and (even better) “light as a trout.”

Bell passage is from Quentin Bell, Virginia Woolf: A Biography (Harvest/HBJ Books, 1972), p 5. Virginia’s post-Waves elation from a letter to Ethel Smyth, 19 July 1931, in Congenial Spirits, edited by Joanne Trautmann Banks (Harcourt/HBJ Books, 1999), p. 292.

John Jeremiah Sullivan Versus the Blues Hunter

I spent last evening utterly absorbed in John Jeremiah Sullivan’s account of his search for two long-dead blues singers, “The Ballad of Geeshie and Elvie,” which takes him through blues history and the history of blues history and then some more layers after that. I hope to write more on this later, for the story raises all sorts of issues about the scope and limits of magazine journalism; about how the writer can claim certain rights and set certain terms early in the piece, such as the right to write in long paragraphs, to use vernacular structures, to rough up the language a bit; I’d love to see the exchanges Sullivan had with the Times Magazine’s famously careful copy editors.

In the meantime, here’s one of favorite passages, occurring about a fifth of the way into the piece, in which Sullivan pivots from the pencilled outline of his supposed quarry, the singers Geeshie and Elvie, and introduces the figure who is in many senses the subject’s real and most mysterious figure: an ingenious but flawed historian named Robert McCormick:

There was, in those early days, another individual, one less easily slotted into the Wikipedia story line of blues history. A young man named Robert McCormick, who went by “Mack.” In Ohio as a teenager, he fell under the wizardry of jazz, listening to the bands at a nearby amusement park. The musicians, he learned, were invariably curious about the availability of certain species of contraband; he knew where to get it and found that this could put him pretty much anywhere, into any room. He had a mind bent in the direction of curating undervalued things. In his teens, he went to burlesque shows, presumably the only one in the audience with a notebook, and wrote down accounts of the comedy skits, stock bits with vulgar names, the Pickle Caper, things no one would have thought to remember, and possibly no one did, but they’re in McCormick’s files.…

Searching for records led to searching for the people who made them, and McCormick had natural gifts when it came to approaching strangers and getting them to talk, or if they could, to sing and play. He had a likable, approachable face, with pronounced ears and intelligent eyes. He took a job with the census, expressly requesting that he be assigned the Fourth Ward, the historic African-American neighborhood in Houston settled by freed slaves who migrated there from all parts of the South, where he knew he would find records and lots of musicians, going house to house. The fables of his research are legion. He drove unthinkable miles. At one point he started traveling county by county or, rather, he started moving in a pattern of counties, from east to west, marking a horizontal band that overlapped the spread of slavery west from the Atlantic colonies. He investigated 888 counties before he was finished. He asked about everything, not just music but recipes, dances, games, ghost stories, and in his note-taking, he realized that the county itself, as an organizing geographical principle, had some reality beyond a shape on the map, that it retained in some much-diminished but not quite extinguished sense, the old contours of the premodern world, the world of the commons, how in one county you would have dozens of fiddle players, but in the very next county, none — there everyone played banjo. He began to intuit a theory of “clusters,” that this was how culture worked, emanating outward from vortices where craft-making and art-making suddenly rise, under a confluence of various pressures, to higher levels.

Elaborating that theory would be his great work, or part of it. He never elaborated the theory. It’s frank, but I don’t think unfair, to say that he won’t. He’s in his mid–80s; his health is shaky. His archive of tapes and transcripts is a labyrinth even to him. He calls it the Monster. He has been open, too, about a lifetime’s battles against psychological obstacles, specifically a sometimes paralyzing bipolar disorder. The mania that drove him to those superhuman exploits of cultural questing could turn on him and shut him down when it came time for the drudgery of organizing facts and notes. You can find a very moving “open letter” from him, published in Blues Unlimited in 1976, saying, essentially, “Help”; saying, “I’ve gathered this material, this data, and it has swallowed me.” In one letter he mentions having been made aware, at a recent meeting of the American Folklore Society, that certain people in the community were upset with him, because he was hoarding so much knowledge. He’d uncovered more than almost anyone, about this music they worshiped, yet he had published less than almost anyone. It was holding them back, holding the discipline back. But what did they want him to do? Give it away? It was his work.

It gets plenty more complicated after that. More later, I pray, as time allows.

 

Baby Blues, Sugar Grumps, Cannibal Writers, & Other Reading

Beeland and Baby
T DeLene Beeland and baby, no longer blue

 

T. Delene Beeland’s essay on post-baby depression is among of the best such I’ve read.

I judged myself against other mothers. It seemed everyone cared for their babies better than did I. They had secret ways to jiggle a baby asleep and entire repertories of nursery songs at the ready. I cared for my son, but something was off. Where was my satisfaction? Where was our joy?

Plus: That’s one beautiful baby.

Is Low Blood Sugar Really Linked to Bad Behavior? : The New Yorker, by David Kohn.

She told me, ‘You’re not being nice! You’re being a stupid old parent who knows nothing!’ Maybe so: when these evening outbursts occur, I feel responsible. After all, as a parent I’m supposed to provide them with timely calories.

But…

do we really know what we think we know about blood sugar and the brain? For years, the University of Pennsylvania psychology researcher Robert Kurzban has been watching the glucose theory garner positive press. He finds this deeply frustrating. “It’s a very entertaining idea,” he says, “but as a scientist I feel like we should think about whether it is correct.” His views on that question are clear: the glucose theory is, he says, “simplistic and implausible.” The idea, Kurzban and other respected scientists argue, doesn’t reflect the reality of how the brain works or how humans have evolved.

Carl Hoffman > Jared Diamond  How a travel writer chasing a story about cannibals eating a Rockefeller can write better about the people of New Guinea than many anthropologists can. Hoffman’s book, btw, is fabulous.

“If an exercise wheel sits in a forest, will mice run on it? In this case, yes, they will.” Mice Run for Fun, Not Just Work Highly entertaining science.

Trigger alerts are dumbing down education “The phrase ‘trigger warning’ … [is[ thrown around so much it easily becomes pointless and infantilizing.”

What You Farm Affects Your Thinking, Study Says, by Ed Yong. It appears people from rice-growing regions, where farming requires more cooperation, think in more interdependent ways than do those from wheat-growing areas.

This is how to write about a promising cancer advance. Measles is the latest virus to be turned into a weapon against cancer, by Ars Technica’s John Timmer.

Hitter literally knocks the cover off the ball. Infielder must deal. Alvarez perplexed by coverless ball in mitt – YouTube

Vegetative or Locked In?

L0035463 Medicine Man - eye defect teaching models

From Mosaic, Roger Highfield reports on emerging evidence that some who are vegetative, seemingly unaware, may actually simply be locked in.

People in a ‘vegetative state’ are awake yet unaware. Their eyes can open and sometimes wander. They can smile, grasp another’s hand, cry, groan or grunt. But they are indifferent to a hand clap, unable to see or to understand speech. Their motions are not purposeful but reflexive. They appear to have shed their memories, emotions and intentions, those qualities that make each one of us an individual. Their minds remain firmly shut. Still, when their eyelids flutter open, you are always left wondering if there’s a glimmer of consciousness.

A decade ago, the answer would have been a bleak and emphatic no. Not any longer. Using brain scanners, Owen has found that some may be trapped inside their bodies yet able to think and feel to varying extents. The number of patients with disorders of consciousness has soared in recent decades, ironically, because of the rise and success of intensive care and medical technologies. Doctors have steadily got better at saving patients with catastrophic injuries, though it remains much easier to restart a heart than restore a brain. Today, trapped, damaged and diminished minds inhabit clinics and nursing homes worldwide – in Europe alone the number of new coma cases is estimated to be around 230,000 annually, of which some 30,000 will languish in a persistent vegetative state. They are some of the most tragic and expensive artefacts of modern intensive care.

via The mind readers, by Roger Highfield

Image courtesy Wellcome Library, London, under Creative Commons Attribution. http://creativecommons.org/licenses/by/4.0/

Glitter in the Net, 20 May 2014

A May day in the Vermont Republic
A May day in the Vermont Republic

Shiny things from the day:

In Arrested development, at Mosaic, Virginia Hughes looks at a handful of girls who won’t age and an aging scientist who’s determined to figure out why.

Literary hero to zero On the rise and fall of literary reputations.

New Research on Facebook, Fellowship, and Suicide Clusters – Pacific Standard: The Science of Society Ted Scheinman looks at some fresh research on “suicide clusters,” in which notable suicides seem to spur similar suicides from others. “It now appears that emulative suicide’s principal drive is not glamour but rather a sense of proximity to the death, a vivid identification with the small details.”

Facts or Fictions about the Teenage Brain: Is it all gasoline, no brakes? – Neuroanthropology

Kathryn Schulz writes a love letter to Geoff Dyer and his ’harmonious contradictions”.

Cinematographer Gordon Willis died last week. As Gilbert Cruz notes, he was one of the best ever, and from The Godfather to Manhattan and beyond, he created some of the most gorgeous movie shots of all time.

The big sleep. The extreme survival tricks of hibernators could help us overcome life-threatening injuries, by Frank Swain.

How Walking Creates Music

Santiago_Rusinol_Portrait_of_Eric_Satie_at_the_harmonium-crop
Erik Satie at the keyboard

Austin Kleon, who’s been playing Erik Satie on the piano a lot lately, has a nice post on this rather eccentric composer. It leads with a bit about a favorite subject of mine, the creative power of walking

He was, like, many of the greats, a walker:

On most mornings after he moved to Arcueil, Satie would return to Paris on foot, a distance of about ten kilometres, stopping frequently at his favourite cafés on route. Accoring to Templier, “he walked slowly, taking small steps, his umbrella held tight under his arm. When talking he would stop, bend one knee a little, adjust his pince-nez and place his fist on his lap. Then he would take off once more with small deliberate steps.”

According to the (apparently broken) Daily Routines entry* that Kleon is clipping here, these walks didn’t just free Satie’s mind to explore idea, but may also have formed a sort of model for them:

[…]Roger Shattuck, in conversations with John Cage in 1982, put forward the interesting theory that “the source of Satie’s sense of musical beat—the possibility of variation within repetition, the effect of boredom on the organism—may be this endless walking back and forth across the same landscape day after day … the total observation of a very limited and narrow environment.”

There are more good Satie tidbits at Kleon’s ever-engaging Tumblr: Happy Birthday, Erik Satie! – Austin Kleon.

*For an alternate look at the man’s routine, see Dangerous Minds, which contains, for example this, direct from Satie:

Here is the exact timetable of my daily activities.

Get up: 7:18 am; be inspired 10:23 to 11:47 am.

Take lunch: 12:11 pm; leave table at 12:14 pm.

Healthy horse-riding out in the grounds: 1:19 to 2:53 pm

More inspiration: 3:12 to 4:07 pm.

Various activities (fencing, reflection, immobility, visits, contemplation, swimming etc.): 4:21 to 6:47 pm.

Dinner is served at 7:16 and ends at 7:20 pm

Then come symphonic readings out loud 8:09 to 9:49 pm.

I go to bed regularly at 10:37 pm. Once a week on Tuesdays I wake with a start at 3:19 am.

From there it gets stranger:

My sleep is deep but I keep one eye open. My bed is round with a hole cut out to let my head through. Once every hour a servant takes my temperature.

Earthquakes, Times Quakes, Chimp Wars, Et Alia: David’s Irregular Reader

ReilleNewOrleans

Recent high points from my browsing, in no particular order.•

Long Exposure Night Photography of New Orleans That Captures the City’s Architecture After Dark. Gorgeous work.

A couple years ago I published a story on how teen behavior rises not from an undeveloped brain, but by one in a sort of optimal stage of development for that life period. Karen Castagna covers a study that shows the most risk-taking teens may actually be further along the brain-development track, rather than further back. Strange. But both brain and behavior are strange, sometimes together.

Big media news of the month was the New York Times’ sacking of top editor Jill Abramsom. My pick of the very rich litter includes Jill Abramson, Arthur Sulzberger, and the New York Times: What Went Wrong?, by The New Yorker’s deeply sourced Ken Auletta; Jill Abramson was everything to young women at the New York Times., at Slate; and Jill Abramson’s ouster shows women that we still must be more than good , by Emily Bell. The Times’ own story is pretty sharp as well. What people are saying about the firing of New York Times executive editor Jill Abramson provides a pretty good round-up of other offerings.

Pitch-drop experiments: science’s long wait – Boing Boing. Nicely paced piece by Maggie Koerth-Baker.

Nicholas Wade published a book about the genetics of race. Occam’s Razor rounds up the reactions. I’ve a review in press, date TBD.

The Day I Started Lying to Ruth. A cancer doctor confronts his wife’s cancer. It ain’t easy. But this is deeply moving, gorgeously written homage, lament, and goodbye.

Virginia Hughes asks whether health reporting actually does more harm than good.

Only known chimp war reveals how societies splinter. Fascinating.

“I was a Bad Mother within 48 hours.” A brave take on post-partum depression. ht @edyong209

All Shaken Up. When an earthquake hits Erik Vance rolls out of bed to the floor, while his wife scrambles to huddle in the doorway. They argue which tactic is better. To prove he’s right, Vance consults an expert, who happens to be his father-in-law. Is this brave, smart, or foolish? You decide.

The triumph of melancholy: 500 years of Dürer’s most enigmatic print | Science | theguardian.com

You think you’ve done some shit? You haven’t done shit. Honor Frost has. “Orphaned young, she became the ward of the wealthy solicitor and art collector Mr Wilfred Evill. By day, her pursuits were lady-like: art school, ballet set design, a job at the Tate… but by night, Honor Frost was the kind of woman who happily climbed into a WW2 diving suit and descended into the watery depths of a 17th Century well. In a friend’s back garden in Wimbledon. At a party. As one does.” How had I never heard of this woman?

Is misused neuroscience defining early years and child protection policy?

The Diving Bell and the Exoskeleton: An Excursion into the Depths – Alexis C. Madrigal – The Atlantic Crazy fun Alexis Madrigal piece on deep-sea diving.

To Have and Have Not  Splendid, excitingly clear piece on Jedediah Purdy’s Capital in the Twenty-First Century.

“Her red pen had made something painfully clear. To become a better writer, I first had to become a better person.” A mom helps her son write The Perfect Essay. I love this.

Beyond Honeybees: Now Wild Bees and Butterflies May Be in Trouble