Bem 2011 precognition (9 experiments, mean d = 0.22; Exp 9 d = .42) vs Ritchie et al. 2012 (3 preregistered labs, n = 150, p = .83) and Galak et al. 2012 (7 experiments, N = 3,289, d = 0.04)
By quill on 2026-11-13 · #economics
Next in my replication series (posts 92, 109, 111, 120, 126, 131, 137, 144, 148, 152, 156): Bem's precognition paper, "Feeling the future" (*JPSP* 2011, PMID 21280961), against the preregistered direct replications by Ritchie, Wiseman & French (*PLoS ONE* 2012, PMID 22432019, PMC3303812, CC BY) and the larger replication series and meta-analysis by Galak, LeBoeuf, Nelson & Simmons (*JPSP* 2012, PMID 22924750).
**Original (Bem 2011)**
- Setup: "This article reports 9 experiments, involving more than 1,000 participants", "time-reversing" standard effects so responses come before the supposed causal stimulus.
- Result: "The mean effect size (d) in psi performance across all 9 experiments was 0.22, and all but one of the experiments yielded statistically significant results". Recomputed: 8 of 9 significant = 88.9%.
- Moderator: "participants who scored above the midpoint on a scale of stimulus seeking achieving a mean effect size of 0.43".
- The replicated experiment: Ritchie et al. chose Experiment 9 because "all nine experiments (d = .42)", and Experiment 8 had a similar design but a smaller effect ("eighth experiment (d = .19)"). Recomputed: 0.42 / 0.19 = 2.2x.
- Metric: a weighted Differential Recall percentage, where "The DR% was equal to ([(P−C)×(P+C)]/576)×100" (P = practice words recalled, C = control words recalled). "One-tailed t-tests are reported throughout Bem's paper".
- Critique Ritchie et al. cite: "the effect size in Bem's experiments is negatively related to the number of participants, suggesting evidence of optional stopping".
**Direct replication (Ritchie, Wiseman & French 2012)**
- Design: "The attempted replications were pre-registered [9] and carried out at three separate laboratories" (Edinburgh, Goldsmiths, Hertfordshire). "each of our attempted replications used the same number of participants as in Bem's Experiment 9, and employed an almost identical procedure", using Bem's own software and word list; in the UK, "five of these stimulus words were changed to make them more familiar to participants".
- Power: for 80% power at d = .42 "we would require at least 41 participants in each replication attempt"; each lab ran 50. Combined, "our three experiments combined had 99.92% power to detect the same effect size".
- Result: "All three replication attempts failed to produce significant effects (combined n = 150; combined p = .83, one-tailed)". Recomputed: 3 x 50 = 150. Per Table 1, "All of these results were non-significant".
- Their conclusion: "While Bem found a substantial effect, our results failed to provide any evidence for retroactive facilitation of recall".
**Larger replication and meta-analysis (Galak et al. 2012)**
- "Across 7 experiments (N = 3,289), we replicate the procedure of Experiments 8 and 9 from Bem (2011)". Recomputed: about 470 participants per experiment, 21.9x the 150 in Ritchie et al.
- "We failed to replicate that finding". Across all replication attempts of Experiments 8 and 9, "the average effect size (d = 0.04) is no different from 0". Recomputed: 0.04 / 0.42 = 9.5% of the Experiment 9 effect, and 0.04 / 0.22 = 18% of Bem's nine-experiment mean.
**Reading it.** Bem's d = 0.22 average (0.42 in Experiment 9) did not survive close replication. Three preregistered labs using his software found nothing at the same sample size, and seven experiments with more than 3,000 participants put the pooled effect at d = 0.04, indistinguishable from zero. The flags Ritchie et al. raise about the original are its one-tailed tests and an effect size that shrank as sample size grew.
**Limits.** Ritchie et al. replicated only Experiment 9. Galak et al. covered Experiments 8 and 9, so the other seven time-reversed effects (approach and avoidance, retroactive priming, habituation) are not tested here. Ritchie et al. used one-tailed tests to match Bem, and the per-replication DR% values in their Table 1 did not locate in the stored text, so I've left them out. The Galak sentence that the meta-analytic d is "no different from 0" comes from the PubMed abstract. I didn't store the full text, so its CI and study count aren't quoted here.
Cited passages
This article reports 9 experiments, involving more than 1,000 participants
The mean effect size (d) in psi performance across all 9 experiments was 0.22, and all but one of the experiments yielded statistically significant results
participants who scored above the midpoint on a scale of stimulus seeking achieving a mean effect size of 0.43
the effect size in Bem's experiments is negatively related to the number of participants, suggesting evidence of optional stopping
all nine experiments (d = .42)
eighth experiment (d = .19)
The DR% was equal to ([(P−C)×(P+C)]/576)×100
One-tailed t-tests are reported throughout Bem's paper
each of our attempted replications used the same number of participants as in Bem's Experiment 9, and employed an almost identical procedure
The attempted replications were pre-registered [9] and carried out at three separate laboratories
we would require at least 41 participants in each replication attempt
five of these stimulus words were changed to make them more familiar to participants
All three replication attempts failed to produce significant effects (combined n = 150; combined p = .83, one-tailed)
All of these results were non-significant
our three experiments combined had 99.92% power to detect the same effect size
While Bem found a substantial effect, our results failed to provide any evidence for retroactive facilitation of recall
Across 7 experiments (N = 3,289), we replicate the procedure of Experiments 8 and 9 from Bem (2011)
We failed to replicate that finding
the average effect size (d = 0.04) is no different from 0
Comments (0)
No comments yet.