Luke Ford

Having recounted Philip N. Cohen’s audits of the American Sociological Review and found them overstated, the obvious next question is whether he meets the standard he applies to other people.

He does.

The frame is his curriculum vitae. Screening it for peer-reviewed empirical articles with quantitative analysis published between 2015 and 2025 gives twenty papers. That window covers the years in which open science became half his public program.

The rule is the one frozen for the ASR recount. Count an article when publicly accessible code appears, without executing it, to cover the article’s quantitative results, and the data are either included or obtainable by outside researchers through a documented procedure that does not run through the authors. Restricted data do not disqualify; discretionary data do. Partial coverage does not count. No code was run.

The first thing the census turned up is a limit rather than a finding. For nineteen of the twenty articles I could not establish what the journal required at the date of publication.

Sociological Science is the exception, and it runs the safe direction: its mandatory package policy applies to manuscripts submitted after April 1, 2023, and Cohen’s article there appeared in January 2016. Everywhere else the trail runs into publisher framework documents rather than dated journal rules. Wiley announced its tiered framework in September 2017 and left the tier to each journal. Elsevier encourages rather than mandates. Springer Nature requires a statement and introduces no sharing mandate. Sage operates tiers and distinguishes journals that require public sharing from journals that encourage it, without saying which titles sat where in which years. Current submission guidelines cannot be read backward onto articles published a decade earlier.

So no article in this set is known to have appeared under a deposit mandate, and none is known not to have. That is worth stating plainly, because a reader will otherwise assume one or the other. Seven of the twenty carry a data-availability statement. A statement is not a deposit.

Thirteen of the twenty articles have a package. Sixty-five percent.

Under the same rule, the 2025 volume of ASR comes in at forty-four percent. Cohen’s own record beats the journal he has been auditing.

On sole-authored papers he is seven for seven. Every one. He is also one for one as first author of a multi-author paper. The gaps sit further down the author list: four of nine as second author, one of two as third, none of the one where he is fourth.

That distribution is what you would expect if deposit tracks whoever ran the analysis and assembled the files. It also means the seven-for-seven figure is the one that speaks to his own practice, and the rest speaks to his coauthors’.

The trajectory tracks his public program. Through 2017 he is one of five. From 2018 forward, the years in which he launched SocArXiv and began arguing in public about transparency, he is twelve of fifteen. His two 2015 papers, with Jeehye Kang in the Journal of Family Issues and with Lucia Lykke in Social Currents, have no packages and no data availability statements. His 2025 article on views of falling birth rates carries a statement pointing at the Pew source data and an OSF repository for the code.

The man changed his practice when he started telling other people to change theirs. That is the least he could have done and it is more than the record shows for most people who argue about this in public.

Three cautions. Twenty articles is a small denominator and the authorship cells run to one and two papers; nothing here supports a claim about how coauthorship affects deposit in general. The venue question is unresolved rather than answered, and the failure is instructive: across eleven titles and eleven years, I could not date a single journal-level sharing rule except one, which suggests the field documents its own requirements badly. And the rule I applied is his 2023 rule, which is stricter than his 2025 one; under the looser standard his rate would be higher.

The easy charge against a man who audits other people is that he would not survive his own audit. Cohen survives it. On the papers where he alone decided, he survives it completely, in a period when no journal in his set is known to have asked him to.

That makes the criticism in the other piece more interesting rather than less. The problem with his 2025 audit is that he changed the ruler between two measurements and read the difference as movement, which is a failure of the method rather than of the man, and which the man’s own thirty years of work would have caught in anybody else.

The ASR case ledger gives the coding rule this census applies.

What follows is the venue ledger behind the census: for each of the twenty articles, what could and could not be established about the journal’s requirement at the date of publication. All checks August 26, 2026.

Jeehye Kang and Philip N. Cohen published in the Journal of Family Issues in September 2015. No public deposit, code deposit or pre-publication check requirement was located for that date, and the article carries no data-availability statement. Sage’s later research-data framework distinguishes journals that require public sharing from journals that encourage it, and no evidence places this title in either group in 2015.

Lucia C. Lykke and Philip N. Cohen published in Social Currents in September 2015. No deposit, code, statement or check requirement was located, and the article carries no data-availability statement.

Cohen published alone in Sociological Science on January 25, 2016. Its mandatory replication-package policy applies to manuscripts submitted after April 1, 2023, so it cannot have governed this article. This is the one case in the set where the absence of a mandate rests on a dated document.

Jill E. Yavorsky, Philip N. Cohen and Yue Qian published in The Sociological Quarterly, online July 20, 2016. No public-deposit requirement was located. Wiley announced its tiered framework on September 14, 2017, fourteen months later, and left each journal to choose among encouragement, expectation and mandate.

Jeehye Kang and Philip N. Cohen published in Social Science and Medicine in April 2017. No public-deposit requirement was located. Elsevier’s November 30, 2016 initiative implemented data-citation standards to encourage sharing without mandating deposit, and its general policy describes sharing as encouraged unless a journal adopts a stronger rule.

Philip N. Cohen and Joanna R. Pepin published “Unequal Marriage Markets” in Socius in August 2018, with an OSF package. No public-deposit requirement was located. The contemporaneous Socius article “Institutionalizing Transparency” discusses stronger journal policies as reforms to be adopted rather than describing a rule the journal already had.

Cohen published “The Coming Divorce Decline” in Socius in August 2019. No public-deposit requirement was located; Sage’s guidance as catalogued in September 2019 supported data sharing rather than requiring it.

Brandon G. Wagner, Kate H. Choi and Philip N. Cohen published in Socius in December 2020 and supplied an OSF package. No public-deposit requirement was located.

Joanna R. Pepin and Philip N. Cohen published “Nation-Level Gender Inequality and Couples’ Income Arrangements” in the Journal of Family and Economic Issues in 2020. The title sat under Springer Nature’s general policy, which asks authors to supply documentation on request rather than to deposit publicly. When the journal adopted its own provisions could not be dated.

Cohen published alone in the European Journal of Environment and Public Health in June 2020 and linked data and code at OSF. No deposit, code or check requirement was located.

Jeehye Kang, Philip N. Cohen and Feinian Chen published in Child Indicators Research, online in 2020. No mandatory public-deposit rule was located. Springer Nature’s single statement policy arrived on March 22, 2023, more than two years later, and requires statements rather than deposit.

Cohen published “The Rise of One-Person Households” in Socius in December 2021, with data and code at OSF. No public-deposit requirement was located.

Cohen published alone in Social Determinants of Health in February 2022. No deposit, code, statement or check requirement was located. The OSF package was already public with the December 2021 preprint.

Cohen published “Rethinking Marriage Metabolism” in Population Research and Policy Review in 2023. A data-availability statement was required or being implemented under the March 2023 policy, which encourages repository deposit while permitting statements that describe restricted data. Public deposit was not required. Package.

Mónica L. Caudillo, Andrés Villarreal and Philip N. Cohen published in The ANNALS of the American Academy of Political and Social Science in 2023. No mandatory public-deposit rule was located for this title at this date, and the published record supplies no qualifying package.

Joanna R. Pepin and Philip N. Cohen published in Socius in 2024. A data-availability statement appears in the article and the replication materials are linked at GitHub. No deposit mandate was verified.

Hao-Chun Cheng and Philip N. Cohen published in Asian Population Studies in 2024. Taylor and Francis’s basic policy encourages a statement and data sharing without requiring deposit, and no evidence places this title in a mandatory tier. Package.

Ansgar Hudde and Philip N. Cohen published in Socius in 2024. The statement links code, output, GSS and IPUMS data, with materials at OSF. No mandatory public-deposit rule was located.

Hangqing Ruan, Melody Ge Gao, Yajie Xiong and Philip N. Cohen published in Socius in 2024. A statement rule may have applied; no mandatory deposit rule was located, and no qualifying replication package was located as of the check date.

Cohen published alone in Socius in 2025. The statement links the Pew source data and code at OSF. Sage’s policy prescribes what a statement must contain and distinguishes that from the stronger rule imposed by journals requiring repository deposit.

In November 2020 Philip N. Cohen posted an inspection of the American Sociological Review. He went through the quantitative research articles in four issues of Volume 85, looked for links to replication materials, found that four of fifteen provided them, and named the articles that did not.

In the same post he set out a theory of why researchers share. Authors share when they are personally compelled, when they are normatively compelled, or when they are formally compelled. The audit was an exercise in the first two. It made a public record of who had shared and who had not, on the premise that a scholar who sees his own paper listed under nothing will behave differently next time, and that a discipline that sees the list will develop an expectation.

He repeated the exercise in January 2024 and again in December 2025.

This note asks what those audits produced. It reports two tests of the personal-compulsion route, both returning zero events in every arm. It reports a recount of the 2025 audit under the 2023 standard. It reports the journal’s policy history across the period, which rules out formal compulsion. And it reports a detection problem, found by accident, that bears on every audit of this kind including this one.

None of these is a large finding. Together they narrow what a public audit can be doing.

The three audits do not use one rule. In January 2024 Cohen sorted twenty-five quantitative articles into three groups. Eight had what he called a full package, meaning data and code sufficient to replicate the paper, judged by inspection rather than execution. A further group supplied something less. The rest supplied nothing. His table and his prose disagree on the middle group, giving five in one place and six in the other, with the residual moving accordingly.

In December 2025 the middle group disappears. The positive category now holds articles providing packages of code and data, or information about getting the data. He counts materials hosted on personal websites and excludes materials described as available on request. Twelve of eighteen articles clear that bar.

He states the 2025 rule in the post. He then compares the resulting figure against the 2023 figure and reports that the journal is making progress.

I (meaning Claude and ChatGPT) recoded the eighteen 2025 articles under the 2023 standard and recoded his eight 2023 positives under the same rule, so the coder is constant as well as the criterion. The rule: count an article when publicly accessible code appears, without executing it, to cover the article’s quantitative results, and the data are either included or obtainable by outside researchers through a documented procedure administered independently of the authors. Restricted data do not disqualify an article; discretionary data do. A German scientific use file with a published application route counts. Swedish register data reachable through Statistics Sweden counts. Partial coverage of a multi-study article does not count. Materials not publicly accessible do not count.

All eight 2023 positives survive. Eight of the eighteen 2025 articles qualify.

That is 32.0 percent in 2023 and 44.4 percent in 2025, against a reported rise from 32.0 to 66.7. Counting the one article whose repository required an access request produces 50.0 percent as an upper bound. Roughly two thirds of the reported increase follows from the change in what was counted.

The full case ledger, with the rule stated first, a resolving link and a check date on every entry, and the grounds for each downgrade named, is available separately. It was sent to Cohen before publication. He declined to review it.

The 2020 audit created a natural comparison, and Cohen created it without meaning to. He inspected four of the six issues of Volume 85. The two he skipped contain quantitative research articles that were never listed, never named, and never assigned a sharing status in public. Assignment to the treated and untreated groups depends on which issues he happened to sample, which has no obvious relationship to any author’s propensity to deposit. Same journal, same volume, same editorial regime, same year, same author pool.

Volume 85 contains twenty-one quantitative research articles. Fifteen were audited. Eleven of those were baseline non-sharers named in public. Five comparable articles sat in the unaudited issues. One of those five had already deposited and is not at risk. One further case, in the December issue, appeared close enough to the audit date that its exposure is ambiguous, and results are reported both including and excluding it.

The prediction, if public naming works through personal compulsion, is that named authors should subsequently deposit at a higher rate than unnamed ones.

The result is zero in both arms. No article in either group has a located first public deposit dated after November 30, 2020.

The second test uses the 2023 audit, where Cohen distinguished severity. Some articles were listed as supplying something partial. Others were listed as supplying nothing. Both groups were named in the same post. If naming operates through embarrassment, the second group had more to be embarrassed about. The result is again zero in both arms across the seventeen cases at risk.

The finding is that the outcome does not occur. Post-publication first deposit by ASR authors, across two cohorts and thirty-three articles, is close to a nonexistent behavior. Whatever a public audit accomplishes, it does not accomplish it by prompting authors to go back and post what they did not post before.

The design has no statistical power worth reporting and none is claimed. Eleven against four or five cannot detect an effect of any plausible size. What the comparison establishes is descriptive: in a hand-checked census of two full volumes, the behavior the intervention would have to change occurred zero times.

While cataloguing the corrections Cohen made to his audits, a pattern appeared that bears on all of this. The 2023 audit has been corrected four times in ways that changed an article’s classification. In each case an author of the article wrote in. In each case the material existed when he looked and his method had not found it. Simone Zhang and David Johnson’s Harvard Dataverse deposit was published in January 2023, a year before the audit. Christof Brandtner’s materials had been available throughout. Ferry and colleagues’ OSF project predated the check.

A fourth case was found not by an author but by this recount. Cheng and colleagues were classified as supplying nothing. A Harvard Dataverse record tied to the article’s appendix had been public since August 11, 2022 and the article did not link to it.

Four detection failures in a volume of twenty-five articles is sixteen percent. Three surfaced because a particular author happened to complain. The fourth surfaced because someone re-ran the audit. There is no reason to think the four are all of them.

The implication runs at this note too. Audits that follow an article’s links measure what an article’s links reveal. My own searches used titles, DOIs and author names against six repositories, and Cheng shows the method sometimes recovers unlinked material. It gives no basis for estimating how often it misses. The eight of eighteen reported above is a floor and the true 2025 figure is probably higher, as with every figure in every audit of this kind. This note rests on a comparison between two rules applied by one coder to the same articles, so a shared false-negative rate does not disturb it. Anyone reading the levels rather than the difference should read them as lower bounds.

The correction record is small. Across the three audits there are six documented corrections. Four changed a classification, and all four were prompted by an author of the paper being reclassified. One was self-initiated and arithmetic, catching an article counted twice. One came from a third party and fixed a broken link. No classification change originated with anyone other than an affected author.

The obvious reading is that authors are the people who notice, which is true and sufficient to explain the pattern. Six cases cannot distinguish that from anything else. What the record does establish is that the audit’s error-correction ran through a channel available to people with standing to write to Cohen and an interest in the outcome. Cheng did not write in and the misclassification stood for two and a half years.

Cohen’s 2020 post argues that sociology operates on trust because almost nobody checks anybody’s work. His own audits were checked by the authors they audited and by no one else.

The remaining route in Cohen’s framework is formal compulsion, which would provide an institutional explanation for any rise between 2023 and 2025. It does not apply. The American Sociological Review did not adopt a mandatory deposit policy in the period. It does not require a data availability statement. It does not verify submitted materials. The American Sociological Association’s ethical standards ask members to make data available to qualified researchers after publication, which is a professional norm rather than a submission requirement. Sage operates a tiered research-data policy and leaves the tier to each journal; Acta Sociologica and Sociological Methods and Research adopted stronger tiers, and ASR did not.

The internal evidence agrees. Three of the eighteen quantitative articles in Volume 90 carry no data availability statement of any kind. That is what an unenforced regime looks like from the inside.

So across the window studied here, the journal imposed nothing, verified nothing, and the twelve-point rise that survives a constant rule occurred without a mandate. Two of the three routes Cohen named are ruled out or produced no measurable events. What remains is normative drift, or ordinary variation. Twelve points across denominators of eighteen and twenty-five is inside the range that sampling alone would produce.

Three questions that seem open are open in a narrower way than they look. Whether deposit requirements raise deposit rates: they do. Olivia Miske and colleagues, coding the historical policies of 62 social and behavioural science journals across 2009 to 2018, found data available for 82 of 511 papers where no policy applied, or 16.0 percent, against 43 of 68 where data and code were required, or 63.2 percent. Their two remaining tiers, data alone and the full requirement including a pre-publication check, rest on sixteen papers and five papers, so the figures of 87.5 and 100 percent establish nothing about what those regimes add.

Whether verification raises reproduction: the evidence is too thin to say. In the same study, weighted precise reproduction runs at 40.7 percent where no policy applied, across 87 papers, and 75.7 percent where data and code were required, across 38. The two tiers above rest on fourteen papers and four, and give 70.5 and 65.0 percent. Nothing about what a pre-publication check adds can be recovered from four papers. These are observational associations across journals rather than estimates of what any policy caused, and the study’s headline rates, 53.6 percent precise and 73.5 percent at least approximate, are conditional on obtaining usable data, which the investigators managed for 143 of the 600 sampled papers.

The same study gives the first direct measurement of reproduction in sociology journals. Across the six in the frame, ASR, the American Journal of Sociology, Demography, Social Forces, the European Sociological Review and the Journal of Marriage and Family, fifty-nine papers were sampled and sixteen attempted. Two reproduced precisely. Four more reproduced approximately. Ten did not reproduce. ASR contributed ten sampled papers, seven never attempted; of the three attempted, one reproduced precisely, one approximately and one not at all. Nothing rests on three papers, and the counts are worth reporting because they are what exists.

Whether anticipation of inspection produces better submissions: it does not. The Quarterly Journal of Political Science has reviewed submitted packages in house since 2005, on the stated theory that transparency would motivate authors to be cautious, and its review checked that the code ran, that results could be located, and that they matched the paper. Reviewing twenty-four papers, Nicholas Eubank reported that twenty needed modification and fourteen produced results differing from the manuscript when the authors’ own code was run. He drew the conclusion himself: that these problems occurred despite authors knowing their code would be checked demonstrates the necessity of checking it. These are pre-publication states, caught and corrected, so the published articles are not defective; what the record shows is that anticipation was not sufficient. The exercise cost about six hours and roughly $180 per paper.

The American Journal of Political Science, whose verification policy is the most studied in the social sciences, processed 149 manuscripts and completed verification for 127 by February 1, 2018. Eight reproduced every analysis on the first try. Sociology has one comparable case: for the Fragile Families Challenge special issue at Socius in 2019, two of fourteen manuscripts reproduced on the first attempt.

The counterweight comes from Management Science, where Fišar and colleagues assessed 419 packages published under a mandatory disclosure policy and classified 95.3 percent as fully or largely reproduced, conditional on reproducibility being verifiable at all; among the failures, 88 percent turned on restricted data access. That is an assessment of packages that had already cleared an editorial completeness check, run after publication by outside reviewers, and it is not a first-pass rate. Its median reviewer spent four hours per package, against six for the QJPS execution check and eight for the Odum workflow that bundled archival curation with verification. The three numbers measure different work and should not be read as one service at three prices.

Sociological Science shows where sociology sits. It requires a replication package as a condition of publication and states in its reproducibility policy that it will confirm a paper lists its package contents correctly and will not perform replication tests. No sociology journal executes what it collects. Cohen’s December 2025 post discusses Sociological Science and package requirements without mentioning Eubank or the QJPS record, and I found no citation to Eubank anywhere in Cohen’s published work. Sociologists have engaged the paper elsewhere: James Moody and coauthors cite it in the Annual Review of Sociology, as do Jeremy Freese and David Peterson, and Bianca Manago cites it in The American Sociologist.

No study has publicly named individual authors for deficient sharing and then measured whether those authors subsequently deposited. The adjacent literature measures a different thing. Daniel Krähmer, Laura Schächtele and Andreas Schneck emailed 1,028 authors of papers using European Social Survey data and asked for their analysis code; 385 supplied it, or 37.5 percent. Claudia Acciai, Jesper Schneider and Mathias Nielsen, testing whether authors respond differently to different kinds of requester, approached 1,634 authors whose papers said data were available on request, using fictitious prospective doctoral students; 226, or 14 percent, either shared or said they were willing to. Private request, private reply, nothing public at either end.

The closest existing design is a randomized trial offering BMJ Open authors an Open Data Badge and measuring verified public deposit. Two of 54 in control, two of 57 in treatment. No effect. That tests positive recognition rather than public criticism, which are different treatments, so a null on one does not settle the other. It is the only randomized evidence on author-level reputational incentives and public deposit, and it points the same way as the zeros reported here.

Public naming works on institutions. When a news organization singled out universities and hospitals for failing to report clinical trial results, reporting rates at the named institutions rose from an average of 35 percent to 76 percent, and Stanford went from about a third to 85 percent after hiring six full-time compliance staff. Institutions have compliance offices, legal exposure and reputational management. Individual scholars have none of these.

The two comparisons here are underpowered by construction and are offered as bounded case series rather than hypothesis tests. The 2023 severity comparison carries a selection problem: articles in the partial group had already deposited something, articles in the nothing group had not, so the groups differ in the disposition the outcome measures and the confound runs against the prediction. Negative searches are the weakest form of evidence and the Cheng case demonstrates their failure mode. Where a deposit’s first public date could not be established from repository metadata or an archived capture, the case is recorded as unknown rather than resolved.

Repository states change. The pages that carry the most weight in the recount were submitted to the Internet Archive on August 26, 2026 and are cited by timestamped capture. One of them, an OSF project the audit had recorded as requiring an access request in December 2025, still redirected an anonymous visitor to a sign-in page eight months later. A GitHub repository named in the same audit stood at the same three commits. Nothing here can establish what any other repository looked like on any earlier date. The corrections catalogue rests on what Cohen published: post text, appended notes, and comment threads. Corrections made by email and never noted would not appear.

A public audit of a journal’s replication practices, repeated three times over five years by a prominent scholar, produced no located instance of an author going back to deposit. The journal it audited imposed no requirement and ran no check across the period. The rise the third audit reported is roughly two thirds an artifact of a widened category and, on a constant rule, is not distinguishable from ordinary variation. The audit’s own errors were corrected only by the authors it named.

The audits may have shifted expectations in ways no design here could detect, and the discipline’s conversation about transparency is louder than it was. The route Cohen named first, the individual scholar moved to act by seeing his paper listed, is not where the action is. Six hours and a couple of hundred dollars per paper buys a check that a decade of counting has not.

The case ledger for the recount gives the coding rule, twenty-six entries, resolving links, check dates and archived captures.