{"id":118150,"date":"2017-10-19T08:07:58","date_gmt":"2017-10-19T16:07:58","guid":{"rendered":"http:\/\/lukeford.net\/blog\/?p=118150"},"modified":"2026-03-28T06:49:16","modified_gmt":"2026-03-28T14:49:16","slug":"andrew-gelman-beyond-power-pose-using-replication-failures-and-a-better-understanding-of-data-collection-and-analysis-to-do-better-science","status":"publish","type":"post","link":"https:\/\/lukeford.net\/blog\/?p=118150","title":{"rendered":"Andrew Gelman: Beyond \u201cpower pose\u201d: Using replication failures and a better understanding of data collection and analysis to do better science"},"content":{"rendered":"

Andrew Gelman writes<\/a>:<\/p>\n

A bunch of people pointed me to a New York Times article by Susan Dominus<\/a> about Amy Cuddy, the psychology researcher and Ted-talk star famous for the following claim (made in a paper written with Dana Carney and Andy Yap and published in 2010):<\/p>\n

\n

That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications.<\/p>\n<\/blockquote>\n

Awkwardly enough, no support for that particular high-stakes claim was ever presented in the journal article<\/a> where it appeared. And, even more awkwardly, key specific claims for which the paper did<\/em> offer some empirical evidence for, failed to show up in a series of external replication studies, first by Ranehill et al. in 2015<\/a> and then more recently various other research teams (see, for example, here<\/a>). Following up on the Ranehill et al. paper was an analysis<\/a> by Joe Simmons and Uri Simonsohn explaining how Carney, Cuddy, and Yap could’ve gotten it wrong in the first place. Also awkward was a full retraction<\/a> by first author Dana Carney, who detailed many ways in which the data were handled in order to pull out apparently statistically significant findings.<\/p>\n

Anyway, that’s all background. I think Dominus’s article is fair, given the inevitable space limitations. I wouldn\u2019t\u2019ve chosen to have written an article about Amy Cuddy—I think Eva Ranehill or Uri Simonsohn would be much more interesting subjects. But, conditional on the article being written largely from Cuddy\u2019s perspective, I think it portrays the rest of us in a reasonable way. As I said to Dominus when she interviewed me, I don\u2019t have any personal animosity toward Cuddy. I just think it\u2019s too bad that the Carney\/Cuddy\/Yap paper got all that publicity and that Cuddy got herself tangled up in defending it. It\u2019s admirable that Carney just walked away from it all. And it\u2019s probably a good call of Yap to pretty much have avoided any further involvement in the matter.<\/p>\n

The only thing that really bugged me about the NYT article is when Cuddy is quoted as saying, \u201cWhy not help social psychologists instead of attacking them on your blog?\u201d and there is no quoted response from me. I remember this came up when Dominus interviewed me for the story, and I responded right away that I have<\/em> helped social psychologists! A lot. I\u2019ve given many talks during the past few years to psychology departments and at professional meetings, and I\u2019ve published several papers in psychology and related fields on how to do better applied research, for example here<\/a>, here<\/a>, here<\/a>, here<\/a>, here<\/a>, here<\/a>, here<\/a>, and here<\/a>. I even wrote an article<\/a>, with Hilda Geurts, for The Clinical Neuropsychologist! So, yeah, I do spend some time helping social psychologists.<\/p>\n

Dominus also writes, “Gelman considers himself someone who is doing others the favor of pointing out their errors, a service for which he would be grateful, he says.” This too is accurate, and let me also emphasize that this is a service for which I not only would<\/em> be grateful. I actually am<\/em> grateful when people point out my errors. It’s happened several times; see for example here<\/a>. When we do science, we can make mistakes. That’s fine. What’s important is to learn from our mistakes.<\/p>\n

In summary, I think Dominus’s article was fair, but I do wish she hadn\u2019t let that particular false implication by Cuddy, the claim that I didn’t help social psychologists, go unchallenged. Then again, I also don\u2019t like it that Cuddy baselessly attacked the work of Simmons and Simonsohn and to my knowledge never has apologized for that. (I’m thinking of Cuddy’s statement, quoted here<\/a>, that Simmons and Simonsohn “are flat-out wrong. Their analyses are riddled with mistakes . . .” I never saw Cuddy present any evidence for these claims.)<\/p>\n

COMMENTS:<\/p>\n

* Steve Sailer: Why has social psychology been the central front in the Replication Crisis?<\/p>\n

I think this is partly because social psychology, as social psychologist Jonathan Haidt has documented, is extremely politicized. On the other hand, it is also because social psychologists are scientific enough to care. Other fields are at least as distorted, but they don\u2019t feel as bad about it as the psychologists do. (At the extreme, cultural anthropologists have turned against science in general: at Stanford, for example, the Anthropology Department broke up for a number of years into Cultural Anthropology and Anthropological Sciences.)<\/p>\n

Is the social psychology glass therefore half empty or half full? I\u2019d say it\u2019s to the credit of social psychologists that they feel guilty enough to host these debates rather than to just ignore them.<\/p>\n

* Andrew Gelman: \u2013 Psychology is a relatively open and uncompetitive field (compared for example to biology). Many researchers will share their data.<\/p>\n

\u2013 Psychology is low budget (compared to biomedicine). So, again, not so much incentive to hoard data or lab procedures. There\u2019s no \u201cRobert Gallo\u201d in psychology who would steal someone\u2019s virus sample in order to get a Nobel Prize.<\/p>\n

\u2013 The financial rewards are lower within psychology, hence the incentive is not to set up your own company using secret technology but rather to get your idea known far and wide so you can get speaking tours, book contracts, etc. Sure, most research psychologists don\u2019t attempt this, but to the extent there are financial rewards, that\u2019s where they are.<\/p>\n

\u2013 In psychology, data are generally not proprietary (as in business) or protected (as in medicine). So there\u2019s a norm of sharing. In bio, if you want someone\u2019s data, you have to beg. In psychology, they have to give you a reason not to share.<\/p>\n

\u2013 In psychology, experiments are easy to replicate (unlike econ or poli sci, where you can\u2019t just run a bunch more recessions or elections) and cheap to replicate (unlike medicine which involves doctors and patients). So replication is a live option, indeed it gets people suggesting that preregistered replication be a requirement in some cases.<\/p>\n

\u2013 Finally, hypotheses in psychology, especially social psychology, are often vague, and data are noisy. Indeed, there often seems to be a tradition of casual measurement, the idea perhaps being that it doesn\u2019t matter exactly what you measure because if you get statistical significance, you\u2019ve discovered something. This is different from econ where it seems there\u2019s more of a tradition of large datasets, careful measurements, and theory-based hypotheses. Anyway, psychology studies often (not always, but often) feature weak theory + weak measurement, which is a recipe for unreplicable findings.<\/p>\n

To put it another way, p-hacking is not the cause of the problem; p-hacking is a symptom. Researchers don\u2019t want to p-hack; they\u2019d prefer to confirm their original hypotheses. They p-hack only because they have to…<\/p>\n

Regarding the issue of why I never contacted Cuddy directly: On the occasions that I have contacted people directly when there have been big problems with their work, I typically have not found such interactions to be useful. Sometimes people don\u2019t respond, other times they seem to miss the point. I do agree that there\u2019s the potential to learn from such a conversation\u2014but there\u2019s also the potential to learn by posting on the blog and getting comments from anyone in the world who might have interest or expertise in the problem.<\/p>\n

What it comes down to, I think, is that there are different styles of interaction. Given that I\u2019ve been blogging daily for over ten years, it\u2019s no surprise that I find blogging to be a useful way of learning from and interacting with people. One reason I started blogging is that it seemed more useful to converse with thousands of people at once, rather than exchanging with people one on one. For some purposes, though, email can be better, and in retrospect maybe this would\u2019ve been one such case. I\u2019m not so sure that Cuddy thinks so, though, given that she never emailed me either.<\/p>\n

Regarding \u201cthe idea of trying to persuade her, in person\u201d: Given that she hadn\u2019t been persuaded by the direct evidence of Ranehill et al., and she hadn\u2019t been persuaded by the very clear arguments of Simmons and Simonsohn, I didn\u2019t (and don\u2019t) see any reason she\u2019d be persuaded by me! After all, I wasn\u2019t really offering any new arguments; my contribution, such as it was, in my blog posts and Slate article (coauthored with Kaiser Fung) was to report the Ranehill et al. and Simmons and Simonsohn articles and add some perspective. So I\u2019m not really sure how the conversation would\u2019ve gone, given that Cuddy had already seen those things and was unpersuaded.<\/p>\n

Just in general I find it easier, and maybe more productive, to present my perspective, address arguments that come in, and consider how I can learn. Direct persuasion rarely works and is stressful, which I guess is what I mean when I said I don\u2019t like interpersonal conflict. Anyway, each of us has our own style of interaction. Here I am responding to blog comments at 5 in the morning, something I don\u2019t usually do!<\/p>\n

* I was also bothered by an implicit claim in Dominus\u2019 article that one has the right to demand their favorite channel of communication.<\/p>\n

* I think that the NYT article made several insinuations that were unjustified, namely that Cuddy facing brutal criticism is driving women out of the field etc. The NYT writer seems to somehow downplay how much Cuddy has benefited from a poorly written piece of science, nor does she point out that Cuddy\u2019s protestations are very much in her own self interest.<\/p>\n

But what\u2019s worse about the article is how it ignores how Cuddy\u2019s actions affect other people in the social sciences. Someone like Cuddy beat out many people\u2013including many women\u2013to receive her tenured job at a top school. That position comes with responsibility, and Cuddy would apparently like to have all the benefits without all of the ensuing challenges. She eagerly took on a public role\u2013no one forced her to do a TED talk\u2013and then became dismayed when she also faced public criticism over it. But it\u2019s all the more appalling considering how people like Cuddy have used shoddy research practices to advance their own careers at the expense of science: that makes it more difficult for those without inside connections to be able to do research and have others pay attention.<\/p>\n","protected":false},"excerpt":{"rendered":"

Andrew Gelman writes: A bunch of people pointed me to a New York Times article by Susan Dominus about Amy Cuddy, the psychology researcher and Ted-talk star famous for the following claim (made in a paper written with Dana Carney … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[42900,16225],"tags":[],"class_list":["post-118150","post","type-post","status-publish","format-standard","hentry","category-andrew-gelman","category-science"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t