{"id":107598,"date":"2016-09-23T07:44:46","date_gmt":"2016-09-23T15:44:46","guid":{"rendered":"http:\/\/lukeford.net\/blog\/?p=107598"},"modified":"2026-03-28T06:48:42","modified_gmt":"2026-03-28T14:48:42","slug":"andrew-gelman-what-has-happened-down-here-is-the-winds-have-changed","status":"publish","type":"post","link":"https:\/\/lukeford.net\/blog\/?p=107598","title":{"rendered":"Andrew Gelman: What has happened down here is the winds have changed"},"content":{"rendered":"

Andrew Gelman writes<\/a>:<\/p>\n

\nTo understand Fiske\u2019s attitude, it helps to realize how fast things have changed.
\nAs of five years ago\u20142011\u2014the replication crisis was barely a cloud on the horizon.<\/p>\n

Here\u2019s what I see as the timeline of important events:<\/p>\n

1960s-1970s: Paul Meehl argues that the standard paradigm of experimental psychology doesn\u2019t work, that \u201ca zealous and clever investigator can slowly wend his way through a tenuous nomological network, performing a long series of related experiments which appear to the uncritical reader as a fine example of \u2018an integrated research program,\u2019 without ever once refuting or corroborating so much as a single strand of the network.\u201d<\/p>\n

Psychologists all knew who Paul Meehl was, but they pretty much ignored his warnings. For example, Robert Rosenthal wrote an influential paper on the \u201cfile drawer problem\u201d but if anything this distracts from the larger problems of the find-statistical-signficance-any-way-you-can-and-declare-victory paradigm.<\/p>\n

1960s: Jacob Cohen studies statistical power, spreading the idea that design and data collection are central to good research in psychology, and culminating in his book, Statistical Power Analysis for the Behavioral Sciences, The research community incorporates Cohen\u2019s methods and terminology into its practice but sidesteps the most important issue by drastically overestimating real-world effect sizes.<\/p>\n

1971: Tversky and Kahneman write \u201cBelief in the law of small numbers,\u201d one of their first studies of persistent biases in human cognition. This early work focuses on resarchers\u2019 misunderstanding of uncertainty and variation (particularly but not limited to p-values and statistical significance), but they and their colleagues soon move into more general lines of inquiry and don\u2019t fully recognize the implication of their work for research practice.<\/p>\n

1980s-1990s: Null hypothesis significance testing becomes increasingly controversial within the world of psychology. Unfortunately this was framed more as a methods question than a research question, and I think the idea was that research protocols are just fine, all that\u2019s needed was a tweaking of the analysis. I didn\u2019t see general airing of Meehl-like conjectures that much published research was useless.<\/p>\n

2006: I first hear about the work of Satoshi Kanazawa, a sociologist who published a series of papers with provocative claims (\u201cEngineers have more sons, nurses have more daughters,\u201d etc.), each of which turns out to be based on some statistical error. I was of course already aware that statistical errors exist, but I hadn\u2019t fully come to terms with the idea that this particular research program, and others like it, were dead on arrival because of too low a signal-to-noise ratio. It still seemed a problem with statistical analysis, to be resolved one error at a time.<\/p>\n

2008: Edward Vul, Christine Harris, Piotr Winkielman, and Harold Pashler write a controversial article, \u201cVoodoo correlations in social neuroscience,\u201d arguing not just that some published papers have technical problems but also that these statistical problems are distorting the research field, and that many prominent published claims in the area are not to be trusted. This is moving into Meehl territory.<\/p>\n

2008 also saw the start of the blog Neuroskeptic, which started with the usual soft targets (prayer studies, vaccine deniers), then started to criticize science hype (\u201cI\u2019d like to make it clear that I\u2019m not out to criticize the paper itself or the authors . . . I think the data from this study are valuable and interesting \u2013 to a specialist. What concerns me is the way in which this study and others like it are reported, and indeed the fact that they are repored as news at all,\u201d but soon moved to larger criticisms of the field. I don\u2019t know that the Neuroskeptic blog per se was such a big deal but it\u2019s symptomatic of a larger shift of science-opinion blogging away from traditional political topics toward internal criticism.<\/p>\n

2011: Joseph Simmons, Leif Nelson, and Uri Simonsohn publish a paper, \u201cFalse-positive psychology,\u201d in Psychological Science introducing the useful term \u201cresearcher degrees of freedom.\u201d Later they come up with the term p-hacking, and Eric Loken and I speak of the garden of forking paths to describe the processes by which researcher degrees of freedom are employed to attain statistical significance.<\/p><\/blockquote>\n

Comments:<\/p>\n

* Yet Fiske doesn\u2019t seem to have any issue with fluffy TED talks. Apparently TED provides the quality control she mentions.<\/p>\n

* Amy Cuddy\u2019s speaker fees are in tier 6\u2013that is, $40,001 and up.<\/p>\n

Yikes. Well, that would create a bit of an incentive\u2026<\/p>\n

* I would say Fiske isn\u2019t using subterfuge\u2013she\u2019s just incompetent (but a full professor at Princeton!). When incompetence is pointed out, she reacts like an academic\u2013she attempts to silence the source or use ad hominem attacks. But here\u2019s the nice thing\u2013she has to do it publicly, rather than pick up the phone (which is the standard method in academic political science). That\u2019s because she can\u2019t pick up the phone to silence you.<\/p>\n

>Look. I\u2019m not saying these are bad people. Sure, maybe they cut corners here or there, or make some mistakes, but those are all technicalities\u2014at least, that\u2019s how I\u2019m guessing they\u2019re thinking. For Cuddy, Norton, and Fiske to step back and think that maybe almost everything they\u2019ve been doing for years is all a mistake . . . that\u2019s a big jump to take. Indeed, they\u2019ll probably never take it. All the incentives fall in the other direction.<\/p>\n

Solzhenitsyn says that when you have spent your life establishing a lie, what is required is not equivocation but rather a dramatic self-sacrifice (in relation to Ehrenburg\u2013https:\/\/en.wikipedia.org\/wiki\/Ilya_Ehrenburg). I see no chance of that happening in any social science field\u2013tenure means Fiske will be manning (can I use that phrase?) the barricades until they cart her off in her 80\u2019s. Thinking machines will be along in 20-30 years and then universities can dismantle the social sciences and replace them with those.<\/p>\n

* Paul Romer, a former academic who now is chief economist at the World Bank thinks that macroecnomics is a science in failure mode, and thinks that this parallels the evolution of science in general. You can read his arguments at:<\/p>\n

https:\/\/www.law.yale.edu\/system\/files\/area\/workshop\/leo\/leo16_romer.pdf<\/p>\n

The gist of it is that economists have cooked up fancy models involving variables that have no measurable counterpart in the real world, and then use these models to draw conclusions that reflect nothing more than the arbitrary assumptions made to identify the model. Not being familiar with the models he criticizes, I can\u2019t assess his claims, but they sound quite plausible. He has been sounding this alarm for quite a while now, and has published numerous papers which you can easily find by Googling the term \u201cmathiness\u201d (which he coined.)<\/p>\n","protected":false},"excerpt":{"rendered":"

Andrew Gelman writes: To understand Fiske\u2019s attitude, it helps to realize how fast things have changed. As of five years ago\u20142011\u2014the replication crisis was barely a cloud on the horizon. Here\u2019s what I see as the timeline of important events: … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[16281,42900],"tags":[],"class_list":["post-107598","post","type-post","status-publish","format-standard","hentry","category-college","category-andrew-gelman"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t