Does it replicate?

Hello, this is a personal project to track where the scientific consensus stands on popular psychological phenomenon.

What is replication? Why does it matter? A core pillar of science is that others can do the same experiment and get the same result. The social sciences are inherently difficult to study because people are difficult. and when modern researchers attempt to replicate their experiments, they are struggling to get the same result. As more attempts fail, it has created the so-called replication crisis, as researchers double-check the foundations of their field. These concepts have spread for decades through textbooks, best-selling books, and especially through social media, while their very existence is in question.

Goal: I want to make it as easy to understand where the scientific consensus stands on these ideas as it was for the initial ideas to spread. I wish this existed when I went googling, so I’m building it. For more detail, see the introductory blog: Does it replicate?

Last updated February 14, 2024. For best viewing, please view this page on desktop.

Phenomenon	Does it replicate?	Rationale for Opinion	Category	Initial Study Name	Initial Study Date	Metastudy	Successful Replication Attempts	Unsuccessful Replication Attempts	Interesting Discussions
Prospect theory	Yes	Evidence is very strong.	Cognitive bias	Prospect Theory: An Analysis of Decision Under Risk	1979	Global Study Confirms Influential Theory Behind Loss Aversion	Replicating patterns of prospect theory for decision under risk		On Hreha On Behavioral Economics
Anchoring	Yes but constrained	Limited to situations which require explicit thinking about the anchor	Cognitive bias	Judgement under uncertainty: Heuristics and Biases	1974	Do Incidental Environmental Anchors Bias Consumers’ Price Estimations?	Investigating Variation in Replicability: A “Many Labs” Replication Project
Loss aversion (part of prospect theory)	Yes but constrained	Real, but must only for material amounts	Cognitive bias	Prospect Theory: An Analysis of Decision Under Risk	March 1979	1. Global Study Confirms Influential Theory Behind Loss Aversion 2. Revise the Belief in Loss Aversion	Replicating patterns of prospect theory for decision under risk	1. Acceptable losses: the debatable origins of loss aversion 2. The Loss of Loss Aversion 3. Loss aversion (simply) does not materialize for smaller losses	1. Jason Collins blog – Kahneman and Tversky’s “debatable” loss aversion assumption 2. Moderating Loss Aversion: Loss Aversion Has Moderators, But Reports of its Death are Greatly Exaggerated (Sci-Hub)
Dunning-Kruger effect	Mixed	The original idea has not universally replicated, but the idea that evaluating skill is itself a skill seems true.	Cognitive bias	Unskilled and unaware of it	1999		How Noise and a Graphic Subverted Understanding Self-Assessment	Reevaluating the Dunning-Kruger effect	1. The Dunning-Kruger Effect Is Probably Not Real 2. The Dunning-Kruger Effect Explained 3. Yes, The Dunning-Kruger Effect Really Is Real 4. A Statistical Explanation of the Dunning–Kruger Effect
Stanford Marshmallow Test	Mixed	The specific notion that SAT scores correlate with the ability to delay gratification did not replicate across cultures and socioeconomic segments.	Behavior	Attention in delay of gratification	1970			Revisiting the Marshmallow Test	The “marshmallow test” said patience was a key to success. A new replication tells us s’more.
Nudges	Mostly no	The effect itself has replicated; however, when applying statistical controls for publication bias to only publish positive results, the effect is potentially small enough to just be noise. However, some nudges prove far more effective in specific domains.	Decision making	Nudge: Improving Decisions About Health, Wealth, and Happiness	2008	The effectiveness of nudging: A meta-analysis of choice architecture interventions across behavioral domains		RCTs to Scale: Comprehensive Evidence from Two Nudge Units	1. No evidence for nudging after adjusting for publication bias 2. Bad News For Nudges
Ego depletion	Mostly no	When the evidence is this mixed, I have to lean no.	Cognitive bias	Ego Depletion and Self-Control Failure	1998	1. Again, No Evidence for or Against the Existence of Ego Depletion 2. Is Ego Depletion Real? 3. Ego depletion and the strength model of self-control 4. An updated meta-analysis of the ego depletion effect
Priming	Mostly no	Fails in many circumstances, such that if it is real, it could only be in specific circumstances.	Decision making	Category accessibility and impression forming	1977			Investigating Variation in Replicability: A “Many Labs” Replication Project
Stanford Prison Experiment	💩	Scientifically, this study was so bad they changed how science is done to prevent its abuses.	Behavior	A Study of Prisoners and Guards in a Simulated Prison	1973			1. The BBC Prison Study (science-ish, still not a real result) 2. Rethinking the nature of cruelty	1. Have results from the Stanford Prison Experiment been replicated by others? 2. Stanford Prison Experiment: why famous psychology studies are now being torn apartx

Got feedback? I welcome any feedback on any topic, including my verdicts, rationales, data sources, and discussions. You can email me at main@replicateornot.com or on Twitter @replicateornot.