3117: Replication Crisis

2025-07-21T05:04:12Z

2603:8000:5E00:2913:CF5:1938:7911:6FA3: /* Transcript */ cat

{{comic
| number = 3117
| date = July 18, 2025
| title = Replication Crisis
| image = replication_crisis_2x.png
| imagesize = 367x492px
| noexpand = true
| titletext = Maybe encouraging the publication of null results isn't enough--maybe we need a journal devoted to publishing results the study authors find personally annoying.
}}

==Explanation==
{{incomplete|This page was created by an UNREPLICATED RESULT. Don't remove this notice too soon.}}

The {{w|replication crisis}} in science refers to the existence of a number of published scientific results that others are unable to reproduce. One aspect of the scientific method is the replication of results, so the failure to replicate some results casts doubt on the validity of the results and scientific knowledge built on them.

Research into the replication crisis itself has been done, with a number of studies being redone and the results compared with the original studies. Many of these redone studies failed to reproduce the results of the original.

In this comic, a research team is looking to see if the situation has improved. Unfortunately as the team had discovered, it had not improved, and in fact they had found many of the same issues as they did in the 2010s.

The fourth panel shows a newspaper stating that the replication crisis has been solved. Given the findings of the team this couldn't be further from the truth. However while the study the team did was not a replication study (such a study would look at the same papers as they did in the 2010s, while this study was likely looking at newer papers), they did get the same result, which could count as a "replication" in scientific reporting (which has a tendency to distort scientific findings, as seen in [[882: Significant]]). This "replication" was then taken to mean that the crisis had been solved.

There is a further possible jab: the replication crises has indeed been "solved", in that the paper authors have shown that the same problems crop up even when scientists are aware of the issue. The "solution" is that the problems persist whether or not the scientists are aware of the replication crisis, so one could simply do science as if the crisis did not happen.

The title text refers to one suggested remedy of encouraging the publication of null results to counteract {{w|publication bias}}, however given the findings of the team this does not appear to have helped. A suggestion is made to create a journal devoted to "personally annoying results", which supposedly would counteract the bias to only publish results that the study authors do not consider "annoying".

The joke is that researchers, being human, are often tempted not to publish results if, for example, the results are not what they were expecting, opposed to a hypothesis they've spoken in favor of, likely to hurt their careers or embarrass them, confusing or difficult for them to explain, or aesthetically or in some other way displeasing to the researcher. Similar to these actual efforts to counteract publication bias, this proposed measure extends this idea, albeit in a way that might sound silly.

[[1574: Trouble for Science]] explores a similar issue.

==Transcript==
{{incomplete transcript|Don't remove this notice too soon.}}

: [Megan, Ponytail, and Cueball are standing at a lectern. Ponytail is talking into the microphone.]
: Ponytail: In the early 2010s, researchers found that many major scientific results couldn't be reproduced.

: [Ponytail turns slightly to look around the room.]
: Ponytail: Over a decade into the replication crisis, we wanted to see if today's studies have become more robust.

: Ponytail: Unfortunately, our replication analysis has found exactly the same problems that those 2010s researchers did.

: [This panel shows a newspaper, with title "NEWS" surrounded by flourishes. There is a photo of the three researches at the lectern taking up about half the front page. The headline reads: "REPLICATION CRISIS SOLVED".]

{{comic discussion}}<noinclude>
[[Category:Comics featuring Megan]]
[[Category:Comics featuring Ponytail]]
[[Category:Comics featuring Cueball]]
[[Category:Scientific research]]

explain xkcd - User contributions [en]

3117: Replication Crisis