Identity Verified Thinker in Business / Industries / Pharmaceuticals
Dennis Lendrem
Dennis Lendrem
Making Data Make Sense

Low Power + False Discoveries

May 7, 2017 4:54 am

Adobe StockUsed only with express written permission

Size Matters in Science when Looking for the Truth

In science, size matters.

As a rule-of-thumb, the power to detect true differences - in experimental treatments, between disease groups, in the control and drug arms of a clinical trial - is greater in larger studies. The larger the study the more likely we are to pick up differences larger than can be simply attributable to chance.

This is more or less true.

Of course, it isn't totally true.

Sometimes increasing the sample size might increase the background variability against which we are measuring any differences. For example, in a clinical trial we may find that the only way to increase the sample size is to widen our trial inclusion criteria. If we do this, we might end up with a more heterogenous study population, increasing variability and reducing the power. But generally speaking, the larger a study the greater the power of the study - the more likely we are to detect genuine differences.

This is well understood.

Less well known is that the smaller the sample size, the more likely we are to obtain a false positive result. In the absence of a true difference - a real difference if you like - the probability of declaring a difference as statistically significant is increased - the false discovery or false positive rate.

The danger is that we end up chasing false discoveries arising entirely by chance.

And, even in the presence of a true difference, small studies overestimate the size of that difference. This is one of the reasons that drugs that looked promising in small Phase 2 clinical trials fail to deliver on that early promise in the critical Phase 3 clinical trials.

There's a useful paper:

And the statistical online forum Cross Validated is useful:

There are currently no comments.
Latest Ebooks