2020: Negative Results

|< < Prev Comic #2020 (July 16, 2018) Next > >|

Negative Results

Title text: P.S. We're going to the beach this weekend, so I'm attaching my preregistration forms for that trip now, before we find out whether it produces any interesting results.

Explanation

Recently, scientists have begun encouraging each other to publish negative results, where a study failed to find the intended effect, as a way of counteracting publication bias (where only interesting positive results get published), which results in false-positive results being published while negative results are not.

Cueball misinterprets the "push to publish negative results" as meaning that he should always attempt to publish the fact that he failed to find evidence of an effect, even when he didn't even try, spending his time playing a video game instead. This plays on the unspoken assumption that scientists would only choose to submit (and journals would only accept) negative results where a study was designed and executed well enough that it should have shown an effect or at least demonstrated evidence of some kind.

Besides personal preferences, The Legend of Zelda: Breath of the Wild, the most recent Legend of Zelda game at the time of publication, was likely chosen for its notable length, Nintendo nerd cred, and a relevance to Nature magazine’s subject. The average time to beat 100% of the content is over 175 hours.

The title text references the practice of "pre-registration" of a study, which is one means to prevent publication bias: details of a planned study are registered with an organization before the study is conducted, so that a null result or a change in methodology cannot be hidden. The title text may be a play on words, mixing this up with registering (or booking) travel. On the other hand, it may just be playing on the absurdity of pre-registering a simple trip to the beach with a registry for scientific studies.

Transcript

[Cueball is sitting in an office chair at a desk typing on a laptop computer. The following message is displayed above him:]

Dear Nature Magazine,

I found no evidence sufficient to reject the null hypothesis in any research areas because I spent the whole week playing The Legend of Zelda: Breath of the Wild.

I'll send you another update next week!

[Caption below the panel:]

The push to publish negative results seems kinda weird, but I'm happy to go along with it.

 add a comment! ⋅  add a topic (use sparingly)! ⋅  refresh comments!

Discussion

The idea of publishing "failed" investigations arose out of the demand to punish all of the results from medical trials. Then there was the realisation that more than one team may have had the same hypotheses, got funding, investigated and not published the proof that they were wrong. So the idea that a A =/= B is still a valuable finding to be reported has come about. There may seem to be lots being published due to years of keeping silent about such results. RIIW - Ponder it (talk) 20:13, 16 July 2018 (UTC)

@RIIW, you meant 'publish' instead of 'punish trial results'? Save the results from violent you! Sebastian --172.68.110.58 05:38, 17 July 2018 (UTC)

There can be several reasons, why a study has a negative outcome or is delayed. Most of those, especially from smaller studies, are not directly related to the matter being investigated, but more to study design, analysis tools or organizational issues. It is much easier to get a wrong or no result than the correct one. The best solution is to somewhere publish these failed experiments and describe the circumstances and reasons so that it can be judged by a third party (even if that is an embarassment for the scientists in an institution). But if you report that you have started a study, and the reasons are rather mundane as in the case within the comic, what should you report? The truth? Should you lie? Sebastian --172.68.110.58 05:38, 17 July 2018 (UTC)

If you start a study and are still working on it, you don't need to publish that you are behind schedule, that's not relevant to the matter being investigated (usually). If you start a study and abandon work on it before you get to the experimental stage, you don't need to be any more specific than that. If, however, you get to the experiments, you should describe them and their results as well as you are able, even if you think that the results are useless or you found that you set the experiments incorrectly ; such results can help someone else who would think about making similar experiments, and in some cases may even be directly useful to someone testing different theory. -- Hkmaly (talk) 22:12, 17 July 2018 (UTC)

I don't think the "null hypothesis" is a reference to 892: Null Hypothesis, as the explanation currently says. Sure, the comic doesn't mention any particular null hypothesis, but it does say "...the null hypothesis in any research areas", which might equally have been phrased "the null hypothesis of any research areas". In which case he's just saying that he hasn't rejected anyone's null hypothesis lately, not that (as in the earlier comic) he's treating "the null hypothesis" as a single, refutable-once-and-for-all thing. -- Peregrine (talk) 08:32, 17 July 2018 (UTC)

The plural in research areas did not seem correct until I read [[1]]a quote from Michael Swan (Practical English Usage, Oxford University Press, 1995):

   "Any is a determiner. It generally suggests an indefinite amount or number, and is used when it is not important to say how much/ how many we are thinking of. Because of its "open," non-specific meaning, any is often used in questions and negative clauses and in other cases where there is an idea of doubt or negation (p. 48)."

Connection to Significant

There was a previous comic about publication bias, 882: Significant. 162.158.89.7 10:02, 18 July 2018 (UTC)