Editing Talk:2755: Effect Size

<!--Please sign your posts with ~~~~ and don't delete this text. New comments should be added at the bottom.-->
Wow, it looks like I'm first![[Special:Contributions/162.158.146.40|162.158.146.40]] 16:40, 27 March 2023 (UTC)

== Wasn't something like this actually done? ==

Robert Sapolsky mentions an obscure paper that actually did something like this.  They did a meta-analysis of the average reported error throughout various disciplines in order of the physical size of the objects being studied (e.g., from cells to organs to etc.), and found no correlation between them.  The conclusion was that this was evidence that philosophical reductionism was flawed.  [[User:Fephisto|Fephisto]] ([[User talk:Fephisto|talk]]) 22:45, 27 March 2023 (UTC)
: Did you manage to find it? [[Special:Contributions/172.70.57.203|172.70.57.203]] 08:49, 28 March 2023 (UTC)
:: [https://www.youtube.com/watch?v=_njf8jwEGRo Here] is the talk.  He talks about the paper around 1:26:00.  The figure is 1:26:50.  [[User:Fephisto|Fephisto]] ([[User talk:Fephisto|talk]]) 13:18, 29 March 2023 (UTC)
:: Maybe [https://muse.jhu.edu/pub/1/article/401203/summary LINK] Titled "Reductionism and Variability in Data: A Meta-Analysis" Sapolsky, R.; Balt S.; Perspectives in Biology and Medicine 39(2), 1996[[User:Tier666|Tier666]] ([[User talk:Tier666|talk]]) 16:21, 29 March 2023 (UTC)

But does the meta-analysis include itself? Technically, it too is part of Science...
Artinum [[Special:Contributions/172.70.91.151|172.70.91.151]] 13:06, 28 March 2023 (UTC)
: It's SCIENCE all the way Down! [[User:Kev|Kev]] ([[User talk:Kev|talk]]) 18:39, 28 March 2023 (UTC)
: I know this is facetious, but to answer seriously, the meta-analysis is a break down of specific areas of science, and meta-analyses was not one of the categories that was analyzed.  [[User:Fephisto|Fephisto]] ([[User talk:Fephisto|talk]]) 14:42, 15 August 2023 (UTC)

scroll box location is ~25.5% down track: scroll box is 10px high, scrollbar is 290px high, 54px above box, 226px below = center of scrollbox is 59/231 = 25.541..% = ~209,815 pages of total studies. Adjusted to 210,000 to account for rounding errors. (Plus the scroll box might not even move a pixel for a number of pages).[[Special:Contributions/162.158.146.41|162.158.146.41]]
: Wait, if the scrollbar is 290px high, then shouldn't the position be 59/290 = 20.345%? It looks a lot more like 1/5th down than 1/4th down to my eyes.  --[[User:Orion205|Orion205]] ([[User talk:Orion205|talk]]) 17:16, 29 March 2023 (UTC)
: The assumption here is that the scroll bar corresponds to the page numbers. However, that is not normally the case, it's more common to have a scroll bar per page, meaning we are here 20% into page nr 53589... -- [[User:Pbb|Pbb]] ([[User talk:Pbb|talk]]) 16:56, 7 April 2023 (UTC)

Did anyone notice the asterisk next to one of the graph elements? There's got to be a lot of those... Not all scientific studies (I would say very few) can be boiled down to a single numerical output.[[Special:Contributions/162.158.146.41|162.158.146.41]]


Unless I misunderstand this, there's also an aspect of this that's due to sign - because some studies of some outcomes expect negative results, and some expect positive, mixing even results that are overall statistically significant may cause the effects to cancel out.
[[User:Mattwigway|Mattwigway]] ([[User talk:Mattwigway|talk]]) 15:32, 28 March 2023 (UTC)
:I think that could be squared[[User:Tier666|Tier666]] ([[User talk:Tier666|talk]]) 17:03, 29 March 2023 (UTC)


meta-analyses are also referenced in [[1477: Meta-Analysis]] [[Special:Contributions/172.71.26.104|172.71.26.104]] 16:18, 28 March 2023 (UTC)Bumpf
: 1477 Is [[1477|Star Wars]]? [[User:Kev|Kev]] ([[User talk:Kev|talk]]) 18:39, 28 March 2023 (UTC)
:: sorry, I meant [[1447: Meta-Analysis]] :) [[Special:Contributions/172.71.166.248|172.71.166.248]] 13:04, 29 March 2023 (UTC)Bumpf
:Would this meta-analysis of all science satisfy Life Goal #28 (assuming it's rejected, as it probably should be)? [[User:Barmar|Barmar]] ([[User talk:Barmar|talk]]) 15:29, 29 March 2023 (UTC)


[[Special:Contributions/172.70.251.39|172.70.251.39]] 07:01, 30 March 2023 (UTC)
'''SCIENCE IS HIGHLY SIGNIFICANT'''

If we (i) postulate that the picture of page 53,589 of the meta-analysis of all science is a representative sample, and if we (ii) postulate that the model of the meta-analysis is just simple random sampling, without stratification (and I think that is a reasonable guess, since if you really have data of ALL science or want to make an assumption about ALL science based on a sample, then Simple random sampling is okay since weighting of different scientific disciplines is proportional to the number of studies in your sample, SRS guarantees getting an unbiased estimate ...), and if we (iii) postulate that the study-specific variance is independent from the single-study means, we can approximately calculate the correct confidence interval.

Let's do it: The authors say that the weighted least square estimator of the population mean is 0.17. The picture shows 11 studies. I eye-balled the effects being (-0.125; 0.5; 0.375; 0.75; -0.375; 3.75; 0.125; 1.25; 0; 0.55; -0,2) and calculated the "between study standard deviation" (using Excel ) being 1.146 and the mean of that sub-sample being 0.6. (Remark: We can ignore the within study variation, since the dominating source of variation is "between studies" and the within error is enclosed in "between study stddev" due to error propagation). Of course, data analysis can be done with a mixed model with clustered data, but doing an analysis with the study means will give a very good approximation.

Now, first step is to calculate the confidence interval of the mean effect size based on the studies we see. We have 11 studies, 10 degrees of freedom. Assuming a t-distribution the (unweighted) 95% confidence interval of the studies in the picture is
0.6 +/- [2.228*1.146/sqrt(11)] = 0.6 +/- 0.77 = [-0.17, 1.37]

The C.I. includes zero but also includes the full meta study mean of 0.17. So, we have no evidence against our hypothesis that page 53,589 which we see on the website is representative for the full meta analysis. So, we can go on 

The 95% confidence interval for ALL studies assuming a number of around 250,000 studies would be
0.17 +/- [1.96*1.146/sqrt(250000)] = 0.17 +/- 0.00572 = [0.16428, 0.17572].

The 99.9% confidence interval for ALL studies assuming a number of around 250,000 studies would be
0.17 +/- [3.3*1.146/sqrt(250000)] = 0.17 +/- 0.00756 = [0.16244, 0.17756].
 
meaning, on average SCIENCE IS HIGHLY SIGNIFICANT (p<0.001)



[[Special:Contributions/162.158.86.191|162.158.86.191]] 10:31, 30 March 2023 (UTC)
I re-viewed the graph and read the comments on the web page. They say the underlying number of papers is 2,3 million. My fault was that I havent multiplied the number of pages with number of studies per page. So, the confidence interval will become even more narrow

The 95% confidence interval for ALL studies assuming a number of around 2,100,000 studies would be
0.17 +/- [1.96*1.146/sqrt(2100000)] = 0.17 +/- 0.00155 = [0.16845, 0.17155].

The 99.9% confidence interval for ALL studies assuming a number of around 250,000 studies would be
0.17 +/- [3.3*1.146/sqrt(2100000)] = 0.17 +/- 0.00261 = [0.16739, 0.17261].

[[Special:Contributions/162.158.86.191|162.158.86.191]] 10:31, 30 March 2023 (UTC)
Interesting is, that the population mean is 0.17 and not 0.000. When averaging the effects of so many studies, all different in topic and investigated treatments and strata, one would expect that the global mean of all effects is zero. But it is 0.17. Clear indication of publication bias. There is higher probability for a positive effect to be published in a paper.