Luke Ford

Andrew Gelman writes:

A bunch of people pointed me to a New York Times article by Susan Dominus about Amy Cuddy, the psychology researcher and Ted-talk star famous for the following claim (made in a paper written with Dana Carney and Andy Yap and published in 2010):

That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications.

Awkwardly enough, no support for that particular high-stakes claim was ever presented in the journal article where it appeared. And, even more awkwardly, key specific claims for which the paper did offer some empirical evidence for, failed to show up in a series of external replication studies, first by Ranehill et al. in 2015 and then more recently various other research teams (see, for example, here). Following up on the Ranehill et al. paper was an analysis by Joe Simmons and Uri Simonsohn explaining how Carney, Cuddy, and Yap could’ve gotten it wrong in the first place. Also awkward was a full retraction by first author Dana Carney, who detailed many ways in which the data were handled in order to pull out apparently statistically significant findings.

Anyway, that’s all background. I think Dominus’s article is fair, given the inevitable space limitations. I wouldn’t’ve chosen to have written an article about Amy Cuddy—I think Eva Ranehill or Uri Simonsohn would be much more interesting subjects. But, conditional on the article being written largely from Cuddy’s perspective, I think it portrays the rest of us in a reasonable way. As I said to Dominus when she interviewed me, I don’t have any personal animosity toward Cuddy. I just think it’s too bad that the Carney/Cuddy/Yap paper got all that publicity and that Cuddy got herself tangled up in defending it. It’s admirable that Carney just walked away from it all. And it’s probably a good call of Yap to pretty much have avoided any further involvement in the matter.

The only thing that really bugged me about the NYT article is when Cuddy is quoted as saying, “Why not help social psychologists instead of attacking them on your blog?” and there is no quoted response from me. I remember this came up when Dominus interviewed me for the story, and I responded right away that I have helped social psychologists! A lot. I’ve given many talks during the past few years to psychology departments and at professional meetings, and I’ve published several papers in psychology and related fields on how to do better applied research, for example here, here, here, here, here, here, here, and here. I even wrote an article, with Hilda Geurts, for The Clinical Neuropsychologist! So, yeah, I do spend some time helping social psychologists.

Dominus also writes, “Gelman considers himself someone who is doing others the favor of pointing out their errors, a service for which he would be grateful, he says.” This too is accurate, and let me also emphasize that this is a service for which I not only would be grateful. I actually am grateful when people point out my errors. It’s happened several times; see for example here. When we do science, we can make mistakes. That’s fine. What’s important is to learn from our mistakes.

In summary, I think Dominus’s article was fair, but I do wish she hadn’t let that particular false implication by Cuddy, the claim that I didn’t help social psychologists, go unchallenged. Then again, I also don’t like it that Cuddy baselessly attacked the work of Simmons and Simonsohn and to my knowledge never has apologized for that. (I’m thinking of Cuddy’s statement, quoted here, that Simmons and Simonsohn “are flat-out wrong. Their analyses are riddled with mistakes . . .” I never saw Cuddy present any evidence for these claims.)

COMMENTS:

* Steve Sailer: Why has social psychology been the central front in the Replication Crisis?

I think this is partly because social psychology, as social psychologist Jonathan Haidt has documented, is extremely politicized. On the other hand, it is also because social psychologists are scientific enough to care. Other fields are at least as distorted, but they don’t feel as bad about it as the psychologists do. (At the extreme, cultural anthropologists have turned against science in general: at Stanford, for example, the Anthropology Department broke up for a number of years into Cultural Anthropology and Anthropological Sciences.)

Is the social psychology glass therefore half empty or half full? I’d say it’s to the credit of social psychologists that they feel guilty enough to host these debates rather than to just ignore them.

* Andrew Gelman: – Psychology is a relatively open and uncompetitive field (compared for example to biology). Many researchers will share their data.

– Psychology is low budget (compared to biomedicine). So, again, not so much incentive to hoard data or lab procedures. There’s no “Robert Gallo” in psychology who would steal someone’s virus sample in order to get a Nobel Prize.

– The financial rewards are lower within psychology, hence the incentive is not to set up your own company using secret technology but rather to get your idea known far and wide so you can get speaking tours, book contracts, etc. Sure, most research psychologists don’t attempt this, but to the extent there are financial rewards, that’s where they are.

– In psychology, data are generally not proprietary (as in business) or protected (as in medicine). So there’s a norm of sharing. In bio, if you want someone’s data, you have to beg. In psychology, they have to give you a reason not to share.

– In psychology, experiments are easy to replicate (unlike econ or poli sci, where you can’t just run a bunch more recessions or elections) and cheap to replicate (unlike medicine which involves doctors and patients). So replication is a live option, indeed it gets people suggesting that preregistered replication be a requirement in some cases.

– Finally, hypotheses in psychology, especially social psychology, are often vague, and data are noisy. Indeed, there often seems to be a tradition of casual measurement, the idea perhaps being that it doesn’t matter exactly what you measure because if you get statistical significance, you’ve discovered something. This is different from econ where it seems there’s more of a tradition of large datasets, careful measurements, and theory-based hypotheses. Anyway, psychology studies often (not always, but often) feature weak theory + weak measurement, which is a recipe for unreplicable findings.

To put it another way, p-hacking is not the cause of the problem; p-hacking is a symptom. Researchers don’t want to p-hack; they’d prefer to confirm their original hypotheses. They p-hack only because they have to…

Regarding the issue of why I never contacted Cuddy directly: On the occasions that I have contacted people directly when there have been big problems with their work, I typically have not found such interactions to be useful. Sometimes people don’t respond, other times they seem to miss the point. I do agree that there’s the potential to learn from such a conversation—but there’s also the potential to learn by posting on the blog and getting comments from anyone in the world who might have interest or expertise in the problem.

What it comes down to, I think, is that there are different styles of interaction. Given that I’ve been blogging daily for over ten years, it’s no surprise that I find blogging to be a useful way of learning from and interacting with people. One reason I started blogging is that it seemed more useful to converse with thousands of people at once, rather than exchanging with people one on one. For some purposes, though, email can be better, and in retrospect maybe this would’ve been one such case. I’m not so sure that Cuddy thinks so, though, given that she never emailed me either.

Regarding “the idea of trying to persuade her, in person”: Given that she hadn’t been persuaded by the direct evidence of Ranehill et al., and she hadn’t been persuaded by the very clear arguments of Simmons and Simonsohn, I didn’t (and don’t) see any reason she’d be persuaded by me! After all, I wasn’t really offering any new arguments; my contribution, such as it was, in my blog posts and Slate article (coauthored with Kaiser Fung) was to report the Ranehill et al. and Simmons and Simonsohn articles and add some perspective. So I’m not really sure how the conversation would’ve gone, given that Cuddy had already seen those things and was unpersuaded.

Just in general I find it easier, and maybe more productive, to present my perspective, address arguments that come in, and consider how I can learn. Direct persuasion rarely works and is stressful, which I guess is what I mean when I said I don’t like interpersonal conflict. Anyway, each of us has our own style of interaction. Here I am responding to blog comments at 5 in the morning, something I don’t usually do!

* I was also bothered by an implicit claim in Dominus’ article that one has the right to demand their favorite channel of communication.

* I think that the NYT article made several insinuations that were unjustified, namely that Cuddy facing brutal criticism is driving women out of the field etc. The NYT writer seems to somehow downplay how much Cuddy has benefited from a poorly written piece of science, nor does she point out that Cuddy’s protestations are very much in her own self interest.

But what’s worse about the article is how it ignores how Cuddy’s actions affect other people in the social sciences. Someone like Cuddy beat out many people–including many women–to receive her tenured job at a top school. That position comes with responsibility, and Cuddy would apparently like to have all the benefits without all of the ensuing challenges. She eagerly took on a public role–no one forced her to do a TED talk–and then became dismayed when she also faced public criticism over it. But it’s all the more appalling considering how people like Cuddy have used shoddy research practices to advance their own careers at the expense of science: that makes it more difficult for those without inside connections to be able to do research and have others pay attention.

Steve Sailer writes:

I’ve written a lot over the years about the Replication Crisis in the social sciences as academics attempt to emulate Malcolm Gladwell’s success on the corporate conference circuit. The New York Times Magazine offers a long sympathetic article about the Power Posing lady at Harvard Business School who has made a lot of money off her claim to have scientifically proven that certain powerful postures will help you get jobs and sales:

When the Revolution Came for Amy Cuddy

As a young social psychologist, she played by the rules and won big: an influential study, a viral TED talk, a prestigious job at Harvard. Then, suddenly, the rules changed.

BY SUSAN DOMINUSOCT. 18, 2017

Blogger Andrew Gelman gets cast in the article as the unsympathetic hardass who hurts the feelings of Dr. Cuddy. (Here’s Gelman responding.)

Dr. Cuddy looks rather like Phoebe on Friends. Granted, that’s not quite the same as looking like Rachel or Monica on Friends, but it’s not bad by academic standards. She probably was not used to being treated in an unladylike manner. She was likely more used to being treated as a cute blonde than as a scientist.

At the end of the article, Dr. Cuddy is getting out of the social science racket and more into writing motivational books and speaking. That’s kind of like how Malcolm Gladwell responded to the barrage of criticism that built up from 2005 onward. That seems reasonable. Some people have a real gift for telling people what they want to hear and they deserve to get paid for it.

COMMENTS:

* One of the subtexts of the article is that all the nasty, bullying, rigid, statistical types were men, and the greatest proportion of their poor, heart-of-gold, victim types were women.

If it weren’t for the Patriarchy and its disgusting mansplaining, power posing would be a real thing, and Cuddy would still be riding high, sharing the stage perhaps with Elizabeth Holmes.

* I haven’t read or heard anything from these TEDTalk social-psych Ph.D.’s that is not covered in the works of charlatan Dale Carnegie or con-man Napoleon Hill.

* One of women’s core competencies is playing by the rules with the goal of getting a trophy at the end without having to actually accomplish something of importance. That is why female students flourish in the educational system, academia in non STEM fields, and highly bureaucratic but non productive fields like HR. It is also why women tend not to flourish in startup companies in complex and rapidly changing fields like tech.

One of the more unintentionally humorous articles in the NYT in recent years was a look at why woman don’t flourish in tech startups. The two feminists found that women are more comfortable with bureaucracy rather than the free flowing, less structured cultures of startups. The proposed solution was for startups to be more bureaucratic to help the ladies. Yeah, that will work.

* If you want to get something done, call a man.

If you want to get someone kicked out, call a woman.

* To nearly all women in present Academia, who have been treated like hothouse flowers all their lives, any criticism constitutes “bullying.” That’s how far we have sunk….

I would add that statistics is based on an incredibly simple concept..Is what we are investigating likely to be real, or is it just an accident due to small data samples? What is the probability of either?

* Whichever sex invests more resources in offspring (usually the female, and pretty much always for placental mammals, given pregnancy and lactation), will see less variation in reproductive success, and cluster around a “safe” strategy, while the other sex, seeing more reproductive variance, compete with one another in pursuit of the higher-investing sex.

Andrew Gelman and Kaiser Fung write:

Consider the case of Amy Cuddy. The Harvard Business School social psychologist is famous for a TED talk, which is among the most popular of all time, and now a book promoting the idea that “a person can, by assuming two simple one-minute poses, embody power and instantly become more powerful.” The so-called “power pose” is characterized by “open, expansive postures”—Slate’s Katy Waldman described it as akin to “a cobra rearing and spreading its hood to the sun, or Wonder Woman with her legs apart and her hands on her hips.” In a published paper from 2010, Cuddy and her collaborators Dana Carney and Andy Yap report that such posing can change your life and your hormone levels. They report that the “results of this study confirmed our prediction that posing in high-power nonverbal displays (as opposed to low-power nonverbal displays) would cause neuroendocrine and behavioral changes for both male and female participants: High-power posers experienced elevations in testosterone, decreases in cortisol, and increased feelings of power and tolerance for risk; low-power posers exhibited the opposite pattern.”

Cuddy’s work on power posing has been covered in the press for years, including in Waldman’s tongue-in-cheek article in Slate. Most of the time, that coverage is glowing. Here’s a recent New York Times review of Cuddy’s new book, Presence: Bringing Your Boldest Self to Your Biggest Challenges: “While Cuddy’s research seems to back up her claims about the effects of power posing, even more convincing are the personal stories sent to the author by some of the 28 million people who have viewed her TED talk. … Unlike so many similar books aimed at ushering us to our best lives, Presence feels at once concrete and inspiring, simple but ambitious—above all, truly powerful.” And here’s a CBS News report from last month: “Believe it or not, her studies show that if you stand like a superhero privately before going into a stressful situation, there will actually be hormonal changes in your body chemistry that cause you to be more confident and in-command. … [M]ake no mistake, Cuddy’s work is grounded in science.”

But the story of power posing is not so simple. An outside team led by Eva Ranehill attempted to replicate the original Carney, Cuddy, and Yap study using a sample population five times larger than the original group. In a paper published in 2015, the Ranehill team reported that they found no effect.

This is not such a surprise. Cuddy’s scientific claim was, as is typically the case, based on finding “statistically significant” results in experiments. We know, though, that it is easy for researchers to find statistically significant comparisons even in a single, small, noisy study. Through the mechanism called p-hacking or the garden of forking paths, any specific reported claim typically represents only one of many analyses that could have been performed on a dataset. A replication is cleaner: When an outside team is focusing on a particular comparison known ahead of time, there is less wiggle room, and results can be more clearly interpreted at face value. The original power-pose study reported an impressively large effect, but that’s what happens with published results from small, noisy studies: Variation is high, so anything that does appear to be statistically significant (the usual requirement for publication) will necessarily be large, even if it represents nothing but chance fluctuation.

New York Times:

Cuddy felt ill when Simmons and Simonsohn published the post with the headline: “Reassessing the Evidence Behind the Most Popular TED Talk.” As illustration, they used a picture of Wonder Woman. Cuddy felt as if Simmons had set them up; that they included her TED talk in the headline made it feel personal, as if they were going after her rather than the work.

The post, which Simonsohn distributed to his email list of hundreds, quickly made the rounds. “People were sending me emails like I was dying of cancer,” Cuddy says. “It was like, ‘We send our condolences,’ ‘Holy crap, this is terrible’ and ‘God bless you; we wish we could do something, but obviously we can’t.’ ” She also knew what was coming, a series of events that did, in fact, transpire over time: subsequent scrutiny of other studies she had published, insulting commentary about her work on the field’s Facebook groups, disdainful headlines about the flimsiness of her research. She paced around, distraught, afraid to look at her email, afraid not to. She had just put together a tenure package and worried that the dust-up would be a continuing distraction.

Cuddy did not like seeing her work criticized in a non-peer-reviewed format, but she wrote a bland statement saying, essentially, that she disagreed with their findings and looked forward to more research on “this important topic.” Carney reassured Cuddy in the months after the Data Colada post that their paper would eventually be vindicated — of course the effects were real; someone would prove it eventually.

Eventually, the Data Colada post caught the eye of another influential blogger, Andrew Gelman, a professor of statistics and political science at Columbia University, whose interest in Cuddy’s work would prove durable, exacting and possibly career-changing for Cuddy. Gelman wields his sizable influence on the field from afar, on his popular blog andrewgelman.com, where he posts his thoughts on best statistical practices in the sciences, with a frequent emphasis on what he sees as the absurd and unscientific. Gelman, who studied math and physics at M.I.T. before turning to statistics, does not believe that social psychology is any more guilty of P-hacking than, say, biology or economics. But he has devoted extensive attention to the field, especially in more recent years, in part because of the way the media has glorified social-psychology research. He is respected enough that his posts are well read; he is cutting enough that many of his critiques are enjoyed with a strong sense of schadenfreude.

Four months after the Data Colada post, Gelman, with a co-author, published an article in Slate about Carney and Cuddy’s 2010 study, calling it “tabloid fodder.” Eventually, Cuddy’s name began appearing regularly in the blog, both in his posts and in comments. Gelman’s writing on Cuddy’s study was coolly dismissive; it bothered him that Cuddy remained fairly silent on the replication and the Data Colada post. For all he knew, Cuddy was still selling the hormone effect in her speaking gigs and in her best-selling book, “Presence,” which he had not read. Had he looked, he would have been annoyed to see that Cuddy did not include a mention of the Ranehill replication. But he might have been surprised to see how little of the book focused on power posing (just a few pages).

On his site, Cuddy’s name, far from the only one he repeatedly invoked, became a go-to synecdoche for faulty science writ large. When he saw that Cuddy had been invited to speak at a conference, he wondered why the organizers had not invited a bunch of other famous figures he clearly considered bad for science, including Diederik Stapel, who had been accused of outright fraud.

His site became a home for frequently hostile comments from his followers. “She has no serious conception of ‘science,’ ” one posted. Another compared Cuddy to Elizabeth Holmes, the Theranos chief executive under investigation for misleading investors. Though Gelman did encourage his readers to stick to the science, he rarely reined anyone in. In one exchange in July 2016, a commenter wrote, “I’ve wondered whether some of Amy Cuddy’s mistakes are due to the fact that she suffered severe head trauma as the result of a car accident some years ago.” Gelman replied, “A head injury hardly seems necessary to explain these mistakes,” pointing out that her adviser, Fiske, whom he has also criticized, had no such injury but made similar errors.

Gelman, whom I met in his office in late June, is not scathing in person; he is rather mild, soft-spoken even. Gelman was vague when asked if he felt there was anything unusual about the frequency of his comments on Cuddy (“People send me things, and I respond,” he said). He said it was Cuddy who was unrelenting. He later emailed me to make sure I was aware that she attacked him and Simmons and Simonsohn on a private Facebook page, without backing up her accusations with evidence; he was still waiting for a clear renunciation of the original 2010 paper on the hormonal effects of power posing. “I would like her to say: ‘Jeez, I didn’t know any better. I was doing what they told me to do. I don’t think I’m a bad person, and it didn’t get replicated’ — rather than salvaging as much as she can.”

Gelman considers himself someone who is doing others the favor of pointing out their errors, a service for which he would be grateful, he says. Cuddy considers him a bully, someone who does not believe that she is entitled to her own interpretation of the research that is her field of expertise…

On Sept. 26, 2016, Amy Cuddy woke up and checked her phone to find a chilling text from a friend. “I’m so sorry,” it said. “Are you O.K.?” She felt a familiar dread, something closer to panic. For the past year, she had mostly stopped going to social-psychology conferences, feeling a chill from her community. Another social psychologist had told her that a graduate student asked if she really was friends with Cuddy. When she responded, “Yes,” the young woman asked, “Why?”

It was the kind of information Cuddy wished she did not have; her closest friends were told to stop passing on or commenting about that kind of thing, but acquaintances still did it. She felt adrift in her field. She worried about asking peers to collaborate, suspecting that they would not want to set themselves up for intense scrutiny. And she felt betrayed, not just by those who cut her down on social media, in blog posts, even in reviews (one reviewer called her “a profiteer,” not hiding his contempt), but also by some of those who did not publicly defend her. She was not wrong to think that at least in some cases, it was fear, rather than lack of support for her, that kept people from speaking up. Two tenured psychology professors at Ivy League universities acknowledged to me that they would have publicly defended some of Cuddy’s positions were they not worried about making themselves targets on Data Colada and elsewhere…

If Amy Cuddy is a victim, she may not seem an obvious one: She has real power, a best-selling book, a thriving speaking career. She did not own up fully to problems in her research or try to replicate her own study. (She says there were real hurdles to doing so, not least of which was finding a collaborator to take that on.) But many of her peers told me that she did not deserve the level of widespread and sometimes vicious criticism she has endured. “Amy has been the target of mockery and meanness on Facebook, on Twitter, in blog posts — I feel like, Wow, I have never seen that in science,” Van Bavel says. “I’ve only been in it for 15 years, but I’ve never seen public humiliation like that.”