{"id":163315,"date":"2025-08-29T05:02:00","date_gmt":"2025-08-29T13:02:00","guid":{"rendered":"https:\/\/lukeford.net\/blog\/?p=163315"},"modified":"2025-08-29T05:10:52","modified_gmt":"2025-08-29T13:10:52","slug":"our-system-often-rewards-bs-rather-than-truth","status":"publish","type":"post","link":"https:\/\/lukeford.net\/blog\/?p=163315","title":{"rendered":"Our System Often Rewards BS Rather Than Truth"},"content":{"rendered":"

Columbia University statistics professor Andrew Gelman blogs<\/a> (here is my collection of his related posts<\/a>): <\/p>\n

Last year we discussed the problem of scientists who host podcasts in which they credulously and uncritically interview celebrity scientists who are promoting junk science. There was Sean Carroll, a physicist who should know better, fawning over<\/a> a Ellen Langer, Harvard psychology professor who was making wild claims about mind-body healing and also uncritically pushing<\/a> the ridiculous claims by Robert Sapolsky, a Stanford biology professor who’s notorious<\/a> for relying on bogus science.<\/p>\n

Both these academic science superstars–the one from Harvard and the one from Stanford–have also been featured uncritically on the Freakonomics podcasts.<\/p>\n

As I wrote a few months ago<\/a>, If you\u2019re a well-trained physicist or economist and you have a public platform and you use it to promote junk science . . . really, what\u2019s the point of it all?<\/p>\n

I mean, really, what’s the point? I can think of three reasons:<\/p>\n

1. You’re invested in the scientist-as-hero narrative (which I hate<\/a>), and these people are NPR and Ted-certified heroes with great stories to tell.<\/p>\n

One reason why these celebrity scientists have such great stories to tell is that they’re not bound by the rules of evidence. Unlike you or me, they’re willing to make strong scientific claims that aren’t backed up by data.<\/p>\n

So it’s not just that Sapolsky and Langer are compelling figures with great stories who just happen to be sloppy with the evidence. It’s more that they are compelling figures with great stories in large part because<\/em> they are willing to be sloppy with the evidence.<\/p>\n

2. Once you have a podcast, you want more listeners. (I have a blog here, I get it.) You get more listeners with good stories. The truth or evidence of the stories is not so important.<\/p>\n

3. You outsource your judgment to the academic community, peer-review process, NPR, Ted, and other podcasts. If someone’s a decorated professor at a top university, with papers published in top journals, further validated by top-grade publicity, then it’s gotta be solid research, right? These science-podcasters are too busy to actually look into the evidence that purportedly supports the wild claims they’re promoting.<\/p>\n

The question then is, what to do about it?<\/p>\n

My original thought was that, if you’re gonna interview people who make outrageous-but-wow-it-would-be-amazing-if-true claims, you should grill them a bit. Express some skepticism and don’t let them just wave away objections.<\/p>\n

The trouble is that if you do this your interview would not go well. If you had me<\/em> on a podcast and asked me tough questions passed along by skeptics who don’t trust Bayesian inference or don’t like polling or whatever, that’s fine–I can respond to such things<\/a>. That would be fine. But if you push hard against people who have the habit of stretching the evidence, I don’t know what would happen. I’m pretty sure they wouldn’t just collapse and admit that their claims are unsupported. My guess is that they’d refer to other studies that they claim would back them up, to which the podcast host would be able to instantaneously respond. So it would just push things back one more step. Either a waste of time or a disaster if the person being interviewed gets angry.<\/p>\n

So I don’t think the strategy of pushing harder in the interview would work.<\/p>\n

I’ve listened to lots of podcasts, and I’ve never heard a single one in which the interviewers really challenge the people being interviewed.<\/p><\/blockquote>\n

Decoding the Gurus are constantly praising Sean Carroll<\/a>.<\/p>\n

Grok: <\/p>\n

This<\/a> is a textbook example of how ignoring design effects in clustered data can inflate confidence. In psych, multilevel data is common (repeated measures, raters), but it’s tricky\u2014easy to get “significant” results from correlated errors. Gelman and Brown emphasize comparing complex models to simple ones (like paired differences) as a gut check, which is solid advice. If the fancy model says one thing and the basic average says another, something’s off….<\/p>\n

Low-power studies + selection bias + hype = unreplicable “wow” effects. Gelman calls it “debasing the currency of discourse”\u2014crowding out real science. He’s right that big-if-true claims (e.g., mindset heals wounds) need extraordinary evidence, but these often rest on p-hacking or misanalysis. The piranha theorem (can’t have tons of large, independent effects coexisting) is invoked, which makes sense mathematically: If every subtle mindset tweak had huge physical impacts, the world would look chaotic.<\/p>\n

Podcasters chase stories, not rigor. Incentives favor “mind-blowing” over mundane. Gelman notes Clarke’s Law (“sufficiently crappy research is indistinguishable from fraud”)\u2014harsh, but when profs like Sapolsky blame journalists for their own exaggerations, it stings. <\/p>\n

Why This Persists: Small n, speculative theories, no pre-reg, plus incentives for big claims. Gelman argues for humility\u2014e.g., frame as “high-risk, high-reward” instead of proven.\n<\/p><\/blockquote>\n

ChatGPT says: The recent spate of controversies around \u201cmind-body\u201d healing studies provides a sharp case study in what Stephen Turner has long argued: that modern science is not just a neutral search for truth but a deeply social enterprise, shaped by institutions, prestige, and professional incentives. When Ellen Langer\u2019s group publishes papers claiming that perceptions of time can alter physical healing, or when Robert Sapolsky casually repeats a fabricated \u201c6000 calories a day\u201d chess-player myth, the problem is not just individual sloppiness\u2014it\u2019s a structural feature of contemporary science.<\/p>\n

As Gelman and Brown show in their forensic re-analysis of the Langer\/Aungle cupping study, what looked like \u201cstatistically significant\u201d findings collapse under scrutiny. The effect sizes wobble, the t-statistics vanish once multilevel variation is properly accounted for, and the literature Langer cites is filled with thinly replicated, often misinterpreted studies. In short: the evidence is not there. Yet the study was published in a legitimate journal, promoted by Harvard, and amplified in popular media.<\/p>\n

Turner\u2019s perspective is especially relevant here. He has long emphasized that what sustains \u201cscientific authority\u201d is not necessarily the reliability of results but the collective trust placed in institutional science. The Langer episode illustrates this clearly: Harvard\u2019s brand, Freakonomics\u2019 enthusiasm, Carroll\u2019s credulous interviewing\u2014all act as amplifiers. The prestige system rewards big, counterintuitive claims (\u201cyou can heal faster if you believe time is different\u201d), while the hard, careful work of methodological critique is marginalized. Science becomes, as Turner might put it, a performance of authority rather than a transparent process of inquiry.<\/p>\n

This performance is sustained by what Turner calls \u201cthe problem of expertise.\u201d In areas where laypeople (and even most scientists) cannot independently evaluate the claims, deference to authority is the default. A physicist like Sean Carroll can interview Langer or Sapolsky without pushing back, because to do so would require specialized statistical knowledge he doesn\u2019t have\u2014or the willingness to publicly puncture the myth of \u201ccelebrity science.\u201d In practice, Carroll\u2019s role is not as skeptic but as conduit. The audience hears \u201cthis is very data-based\u201d, and the institutional trust carries the claim forward.<\/p>\n

Here we see the corrosive effect Turner warns about: when \u201cscientific knowledge\u201d becomes unmoored from rigorous replication and rests instead on networks of authority, prestige, and trust, it functions less like knowledge and more like rhetoric. Junk science is not an anomaly but an outcome of the very structures that elevate \u201cbig thinkers\u201d and \u201cgenius professors.\u201d<\/p>\n

The real damage, then, is not just the spread of dubious healing claims. It\u2019s the erosion of credibility: the debasing of what Gelman calls \u201cthe currency of discourse\u201d. If Harvard and Stanford promote weak or fabricated findings, if journals let methodological errors slip through, if popularizers treat anecdotes as evidence, why should the public\u2014or other scientists\u2014trust these institutions at all?<\/p>\n

Turner would argue that this points to a broader crisis: we cannot rely on \u201cscience as an institution\u201d to guarantee truth. The replication crisis makes clear that institutional filters fail. What\u2019s needed instead is a shift in epistemic culture: away from celebrity amplification, away from spectacular claims, and toward transparency, replication, and humility. Gelman and Brown\u2019s recommendations\u2014replication, preregistration, realistic effect sizes\u2014are steps in that direction, but Turner would add that we must also confront the sociology of authority itself.<\/p>\n

Science cannot escape its social foundations. But if its institutions reward bullshit over rigor, amplification over skepticism, then we shouldn\u2019t be surprised when \u201chealing by mindset\u201d becomes the next frontier of Harvard-backed pseudoscience. The lesson here, seen through Turner\u2019s lens, is stark: the problem is not just bad studies, but the social machinery that makes bad studies authoritative.<\/p>\n

The fiascos around Ellen Langer\u2019s \u201cmind-body healing\u201d studies and Robert Sapolsky\u2019s flamboyant calorie myths aren\u2019t just examples of sloppy science. They expose the deeper pathology Stephen Turner warns against: scientism, the belief that the authority of science extends wherever its rhetoric can reach, regardless of whether the evidence justifies it.<\/p>\n

Scientism thrives on prestige. Harvard or Stanford faculty publish bold, counterintuitive claims\u2014time perception alters healing, chess burns 6000 calories a day\u2014and because the institutions carry weight, the claims are repeated by Freakonomics, NPR, or celebrity podcasters. The underlying evidence collapses on inspection\u2014Gelman and Brown\u2019s re-analysis shows the Langer study\u2019s supposed large effects dissolve into noise once proper modeling is applied. But the structure of scientism ensures that the claim has already done its work: it entered the bloodstream of \u201cwhat science says.\u201d<\/p>\n

Stephen Turner\u2019s critique is that scientism mistakes institutional authority for epistemic warrant. Science, properly understood, is a fragile process of inquiry, full of error and revision. But scientism repackages that messy process into pronouncements delivered with the aura of certainty. It collapses the distinction between \u201cwe have data suggesting X\u201d and \u201cscience shows X.\u201d When Carroll nods along to Langer\u2019s claims with \u201cOh yeah\u201d, he\u2019s not just being a bad interviewer; he\u2019s enacting scientism\u2014affirming that the authority of a Harvard psychologist is enough to settle the matter.<\/p>\n

The damage is twofold. First, scientism encourages bullshit. As Gelman notes, the incentives tilt toward big claims with shaky evidence: that\u2019s what gets you TED talks, book deals, and journalistic fawning. A cautious, modest statement\u2014\u201cchess players may experience stress responses, but caloric expenditure remains unclear\u201d\u2014would never be amplified. Second, scientism corrodes trust. When audiences discover that the grand claims are hollow, they don\u2019t just doubt the celebrity professor; they doubt science itself. The replication crisis shows that this is not paranoia but pattern.<\/p>\n

Scientism, then, is not an overextension of science but a betrayal of it. It treats science as an oracle rather than as inquiry. It thrives on authority, not replication; on spectacle, not method. Turner\u2019s point is that this is not an accident but a structural feature of how modern institutions traffic in \u201cexpertise.\u201d The Harvard name, the physicist interviewer, the popular podcast\u2014these are mechanisms for manufacturing belief, not for scrutinizing truth.<\/p>\n

The way forward is not to double down on scientism\u2014more hype, more trust, more \u201cscience communication\u201d that oversells the weak evidence. It is to accept that science is fallible, limited, and social. To speak honestly about uncertainty. To separate the prestige of institutions from the credibility of specific claims. To refuse the conflation of authority with knowledge.<\/p>\n

In short: the lesson of Langer and Sapolsky is not that \u201cscience sometimes fails,\u201d but that scientism always fails. It mistakes the theater of authority for the substance of inquiry. Turner\u2019s warning is that unless we confront this, we\u2019ll continue to be awash in Harvard-endorsed healing myths and Stanford-fueled calorie fantasies, while the public\u2019s trust\u2014rightly\u2014evaporates.<\/p>\n

The modern scientific enterprise is no longer an archipelago of individual investigators but an institutional complex\u2014journals, universities, funding agencies, and media intermediaries\u2014that generates consensus and distributes credibility. Within such a system, the actual warrant for belief is not the replicability or robustness of results, but the prestige of their institutional carriers. That Harvard psychology faculty or a Stanford biologist have said something is enough to constitute, in practice, what \u201cscience says.\u201d<\/p>\n

This is why a study whose statistical significance evaporates once random effects are modeled correctly, or an anecdotal claim conjured out of numerological error, can nevertheless circulate as fact. Scientism fuses science\u2019s epistemic authority with the social authority of the institutions that speak in its name. The audience cannot, in most cases, assess the methodological details\u2014whether the Langer cupping study\u2019s effect sizes are plausible, or whether Sapolsky\u2019s calorie arithmetic is nonsense. They can only register that \u201cscience has spoken.\u201d<\/p>\n

Turner\u2019s analysis is sharper than the familiar lament that \u201cscience sometimes gets things wrong.\u201d His point is that scientism structurally guarantees such wrongness will be amplified. The economy of prestige rewards spectacular, counterintuitive claims that dramatize the power of mind over body, or the stress of chess as equivalent to elite sport. The institutional and media circuits\u2014journals, TED, Freakonomics, Carroll\u2019s podcast\u2014operate as multipliers of these claims. What matters is not validity but communicability: the production of what Turner calls the performance of expertise.<\/p>\n

This helps explain why skepticism\u2014Gelman and Brown\u2019s meticulous statistical forensics, or critics pointing out the innumeracy of Sapolsky\u2019s calorie math\u2014remains marginal. Their work lacks the institutional glamour and audience. In scientism, critique has no traction because credibility is not adjudicated by the canons of method but by the distribution of authority. To question the prestige-backed claim is to question the institution itself, which the public, journalists, and even scientists in adjacent fields are reluctant to do.<\/p>\n

The result is not merely error but epistemic corruption. Scientism substitutes the aura of certainty for the actual practice of inquiry. It repackages speculative hypotheses as settled science, collapses the distinction between anecdote and evidence, and enacts a vision of science as a secular priesthood\u2014its authority grounded not in the contingency of replication and revision but in the charisma of its institutions and celebrities.<\/p>\n

Turner\u2019s warning, then, is not simply methodological but sociological. The replication crisis demonstrates that even under conditions of good faith, science produces unreliable results at scale. Scientism masks this reality, insisting that institutionalized science is coextensive with truth. The danger is not only that particular claims fail but that the entire currency of epistemic trust is debased.<\/p>\n

To critique scientism, therefore, is to decouple science as practice from science as authority. It is to insist that Harvard or Stanford imprimatur, journal placement, or celebrity endorsement cannot substitute for the arduous, often negative labor of replication and falsification. It is to recognize that the sociology of expertise\u2014the incentives, the prestige hierarchies, the media amplification\u2014systematically biases what becomes \u201cscientific knowledge.\u201d And it is to see that unless this distinction is restored, we are condemned to cycles of enthusiasm, disillusionment, and cynicism.<\/p>\n

Scientism promises certainty where there is only conjecture, authority where there should be doubt. Its collapse is not an aberration but an inevitability. Turner\u2019s contribution is to remind us that what needs reform is not merely statistical practice or publication norms but the very social machinery of scientific authority.<\/p>\n","protected":false},"excerpt":{"rendered":"

Columbia University statistics professor Andrew Gelman blogs (here is my collection of his related posts): Last year we discussed the problem of scientists who host podcasts in which they credulously and uncritically interview celebrity scientists who are promoting junk science. … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[16225,42905],"tags":[],"class_list":["post-163315","post","type-post","status-publish","format-standard","hentry","category-science","category-stephen-turner"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t