Editing 2400: Statistics

{{comic
| number    = 2400
| date      = December 18, 2020
| title     = Statistics
| image     = statistics.png
| titletext = We reject the null hypothesis based on the 'hot damn, check out this chart' test.
}}

==Explanation==
This comic is another in a [[:Category:COVID-19|series of comics]] related to the {{w|COVID-19 pandemic}}, specifically regarding the [[:Category:COVID-19 vaccine|COVID-19 vaccine]]. It is also another one of [[Randall|Randall's]] [[:Category:Tips|Tips]], this time a statistics tip. The next tip comic after this [[2435: Geothmetic Meandian]] had a stats tip.

===Graph===
The main focus of the comic is a graph showing cases of COVID-19 versus time for two groups: one group was vaccinated and the other group was not. Graphs are ways to visualize data, and for real data indicate specific values. This graph seems to be based on [https://www.nejm.org/doi/full/10.1056/nejmoa2034577 the Pfizer vaccine's results]. The higher line ("placebo group") rises in a steep curve. The lower line ("vaccine group") follows the first for a bit but then levels out to a much slower rate of climb. Officially, a scientific assessment of the effectiveness of anything requires rigorous statistical analysis. This is particularly true in medical studies, where impacts of biology can be highly complex and subject to many factors, meaning that careful review of the data is necessary to confirm that an intervention was effective. The joke of this comic is that the intervention presented here is so ''obviously'' effective that it's obvious even to a layman with little understanding of the math. A few days after the vaccine was administered, cases in the vaccinated group essentially flatline, while cases in the placebo group continue to rise as a significant rate. The data is so "good", meaning that numbers for the treatment and control groups diverge so dramatically, that actual analysis becomes almost a formality: a glance at the chart would convince most people that the treatment is effective. 

This comic was released one week after the FDA granted an emergency use authorization for the {{w|BNT162b2|Pfizer COVID-19 vaccine}}, and 8 days after results of its Phase 3 clinical trial were published in [https://www.nejm.org/doi/full/10.1056/nejmoa2034577 the ''New England Journal of Medicine'']. The document includes the following [https://www.nejm.org/na101/home/literatum/publisher/mms/journals/content/nejm/2020/nejm_2020.383.issue-27/nejmoa2034577/20210819/images/img_xlarge/nejmoa2034577_f3.jpeg chart].  The charts draw the integral of the incidence data rather than the data itself ("cumulative" rather than "rate"): this results in changes in disease rate towards the left side of the chart, being added into the data on the right side, amplifying their difference.  This technique for emphasizing the data is valid: the spread between the lines only continues to increase if the effect continues happening, such that the total spread at the right is proportional to the total effect the vaccine had.  The charts do not show any information on other possible variables.  Randall has described previously in his webcomics how very clear charts can be made to hide misleading data.  The linked graph does not leave the numbers out, and the numbers indicate the vaccine is 91% effective at preventing the disease (and a 95% chance of being between 85 and 95% efficient). 

The advice here could be seen as the inverse of the "science tip" in [[2311: Confidence Interval]], in which the data was so ''bad'' that its error bars fell outside of the graph and were not shown. Also there's some association with [[1725: Linear Regression]] where the data is not so good that you don't need to perform linear analysis.

===Null hypothesis===

The null hypothesis, mentioned in the title text, is the hypothesis in a statistical analysis that indicates that the effect investigated by the analysis does not occur, i.e. 'null' as in zero effect. For example, the null hypothesis for this study might be "The vaccine has no effect on whether subjects catch COVID." The null hypothesis was previously the subject of [[892: Null Hypothesis]]. The null hypothesis is rejected when the probability of something like the observed data would be very low were the null hypothesis true.

For a simplified example, imagine there are 10&#8239;000 people in the vaccinated group, and each has a 5% chance of catching COVID under the null hypothesis; we expect 500 people to catch COVID. If only 490 catch COVID, the null hypothesis remains plausible, but if just 10 do, the odds are (in Python; see {{w|binomial distribution}}) <code>sum([math.comb(10000, i) * 0.05**i * 0.95**(10000-i) for i in range(0,10)])</code> = 1.5&nbsp;×&nbsp;10<sup>-204</sup>. In other words, it is wildly improbably that an ineffective vaccine would have produced such excellent results. We therefore conclude that the vaccine is not ineffective, and have rejected the null hypothesis.

Most people however, on seeing the raw results, would have concluded that the vaccine worked and statistics were just a formality. As the title test says, they would have "reject[ed] the null hypothesis based on the 'hot damn, check out this chart' test."

==Transcript==
:[Shown is a graph with the x-axis labeled "time" and the y-axis labeled "COVID cases." There is a black line on the graph labeled "placebo group", which has a roughly linear slope moving toward the top right corner. There is a red line labeled "vaccine group", which follows the black line for about an eighth of the width of the graph before leveling off at a much slower increase.]

:Caption beneath the graph: Statistics tip: Always try to get data that's good enough that you don't need to do statistics on it

{{comic discussion}}


[[Category:COVID-19]]
[[Category:COVID-19 vaccine]]
[[Category:Comics with color]]
[[Category:Statistics]]
[[Category:Tips]]
[[Category:Scientific research]]