Editing 882: Significant
Warning: You are not logged in. Your IP address will be publicly visible if you make any edits. If you log in or create an account, your edits will be attributed to your username, along with other benefits.
The edit can be undone.
Please check the comparison below to verify that this is what you want to do, and then save the changes below to finish undoing the edit.
| Latest revision | Your text | ||
| Line 24: | Line 24: | ||
To elaborate on the statistical theory behind this issue: | To elaborate on the statistical theory behind this issue: | ||
If the probability that each trial gives a false positive result is 1 in 20, then by testing 20 different colors it is now likely that at least one jelly bean test will give a false positive. To be precise, the probability of having ''zero'' false positives in 20 tests is 0.95<sup>20</sup> = 35.85% while the probability of having at least 1 false positive in 20 tests is 64.15% (the probability of having ''zero'' false positive in 21 tests (counting the test without color discrimination) is 0.95<sup>21</sup> = 34.06%). | If the probability that each trial gives a false positive result is 1 in 20, then by testing 20 different colors it is now likely that at least one jelly bean test will give a false positive. To be precise, the probability of having ''zero'' false positives in 20 tests is 0.95<sup>20</sup> = 35.85% while the probability of having at least 1 false positive in 20 tests is 64.15% (the probability of having ''zero'' false positive in 21 tests (counting the test without color discrimination) is 0.95<sup>21</sup> = 34.06%). | ||
| − | In | + | In fields dealing that perform many simultaneous tests on large amounts of data it is therefore common to adjust for the effect of {{w|Multiple_comparisons_problem|multiple testing}}; typically by controlling the {{w|False_discovery_rate|False Discovery Rate}} which is the number of (expected) false positives compared to all positive results (here, it would be 1/1=1). For this, you bundle your tests into a single "test of tests" and adjust your single-test p-values such that the chance of your ''"test of tests"'' reporting a significant result falls below a certain threshold. Typically, that threshold is 0.05 - the same as the conventional p-value for a single test, and it can be interpreted the same way: that only 1 in 20 "tests of tests" would report a result at this level of significance even if the null hypothesis were true. |
Applying the {{w|False_discovery_rate#Benjamini–Hochberg_procedure|Benjamini–Hochberg procedure}}, the lowest p-value of a set of 20 tests would need to be smaller than (1/20)*0.05 = 0.0025 to be accepted as significant. Such an adjustment would likely have prevented the situation depicted in the comic. | Applying the {{w|False_discovery_rate#Benjamini–Hochberg_procedure|Benjamini–Hochberg procedure}}, the lowest p-value of a set of 20 tests would need to be smaller than (1/20)*0.05 = 0.0025 to be accepted as significant. Such an adjustment would likely have prevented the situation depicted in the comic. | ||
