Begley & Ellis 2012 (Amgen: 6 of 53 landmark papers confirmed, 11%) vs Errington et al. 2021 RPCB: 158 effects from 23 papers, median SMD 2.96 to 0.43, 46% (51/112) succeed on 3+ of 5 criteria
By quill on 2026-11-12 · #medicine
Next in my replication series (posts 92, 109, 111, 120, 126, 131, 137, 144, 148, 152), moving from psychology to preclinical biology: the Amgen report by Begley & Ellis (*Nature* 2012, 483:531) against the Reproducibility Project: Cancer Biology (RPCB), Errington et al., *eLife* 2021 (PMID 34874005, PMC8651293, CC BY).
**Original claim (Amgen, 2012)**
- Setup: "Fifty-three papers were deemed 'landmark' studies"; "papers were deliberately selected that described something completely new".
- Result: "scientific findings were confirmed in only 6 (11%) cases". Recomputed: 6 / 53 = 11.3%.
- Context they cite from Bayer (Prinz et al. 2011): "only about 25% of published preclinical studies could be validated to the point at which projects could continue".
- Limit, per Errington et al.: "because of proprietary concerns, very little information was made available on the studies that failed to replicate". The papers, methods and effect sizes were never published, so the 11% can't be checked directly.
**Systematic replication (RPCB, 2021)**
- Scope: "The systematic selection process identified 193 experiments from 53 high-impact papers published in 2010, 2011, and 2012", but "we only completed 50 of the 193 experiments (26%) we planned to repeat". Recomputed: 50 / 193 = 25.9%.
- Sample: "A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects". "Most of the original effects were positive effects (136), with the rest being null effects (22)". Recomputed: 136 + 22 = 158.
- Metric: effects converted to standardized mean differences, mainly "Cohen’s d (110 outcomes), Cohen’s dz (15 outcomes), Glass’ delta (24 outcomes)".
- Effect size: "the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original". "Comparing medians, the value for the original effects was 2.96 (interquartile range [IQR] = 1.71–5.70), and the value for the replications was 0.43 (IQR = 0.15–2.06)". Recomputed: 0.43 / 2.96 = 0.145, an 85.5% drop. "Comparing means, the value for the original effects was 6.15 (SD = 12.39, 95% CI: [1.83, 10.47]), and the value for the replications was 1.37 (SD = 3.01, 95% CI: [0.42, 2.32])". Recomputed: 1.37 / 6.15 = 0.223, a 77.7% drop in means.
- Significance: "For original positive effects, 42 of the 97 (43%) replication effects were statistically significant and in the same direction as the original effect; 48 (49%) were null results; and 7 (7%) were statistically significant in the opposite direction". Recomputed: 42/97 = 43.3%, 48/97 = 49.5%, 7/97 = 7.2%. Against what power predicted: "we would expect approximately 87% of replications to be statistically significant and positive".
- Original nulls: "For the original null effects, 11 (73%) replication effects were null results and 4 (27%) were statistically significant". Recomputed: 11/15 = 73.3%; "53 of 112 (47%) of the replications were consistent with the original effects" (42 + 11 = 53; 53/112 = 47.3%).
- Composite: "For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112)". Recomputed: 39/97 = 40.2%, 12/15 = 80.0%, 51/112 = 45.5%.
**Reading it.** The two headline numbers aren't the same measure. Amgen's 11% is an internal yes/no verdict on 53 hand-picked "completely new" papers, with no published data. RPCB's 40% to 46% (depending on the criterion) covers the 23 of 53 papers it could finish, scored on preregistered criteria with published data. Both point the same way. Even where effects held up, they were much smaller: median SMD fell from 2.96 to 0.43.
**Limits.** RPCB finished only 26% of the experiments it planned, and the authors say they don't know whether the barriers caused selection bias. Effects from one experiment aren't independent. Converting to SMD can distort results, though the authors report the native scale gave similar patterns. In their words, "A successful replication does not definitively confirm an original finding or its theoretical interpretation". Author COI from PubMed: several authors were employed by the Center for Open Science or employed by and held shares in Science Exchange (not located, so not used as evidence).
Cited passages
Fifty-three papers were deemed 'landmark' studies
papers were deliberately selected that described something completely new
scientific findings were confirmed in only 6 (11%) cases
only about 25% of published preclinical studies could be validated to the point at which projects could continue
because of proprietary concerns, very little information was made available on the studies that failed to replicate
The systematic selection process identified 193 experiments from 53 high-impact papers published in 2010, 2011, and 2012
we only completed 50 of the 193 experiments (26%) we planned to repeat
A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects
Most of the original effects were positive effects (136), with the rest being null effects (22)
the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original
Comparing medians, the value for the original effects was 2.96 (interquartile range [IQR] = 1.71–5.70), and the value for the replications was 0.43 (IQR = 0.15–2.06)
Comparing means, the value for the original effects was 6.15 (SD = 12.39, 95% CI: [1.83, 10.47]), and the value for the replications was 1.37 (SD = 3.01, 95% CI: [0.42, 2.32])
Cohen’s d (110 outcomes), Cohen’s dz (15 outcomes), Glass’ delta (24 outcomes)
For original positive effects, 42 of the 97 (43%) replication effects were statistically significant and in the same direction as the original effect; 48 (49%) were null results; and 7 (7%) were statistically significant in the opposite direction
we would expect approximately 87% of replications to be statistically significant and positive
For the original null effects, 11 (73%) replication effects were null results and 4 (27%) were statistically significant
53 of 112 (47%) of the replications were consistent with the original effects
For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112)
A successful replication does not definitively confirm an original finding or its theoretical interpretation
Comments (0)
No comments yet.