Unblinded Studies are Biased Even if the Researchers are Well-Intentioned

Despite a well-known and scientifically accepted practice of using blinded studies to support a claim, it is still not uncommon for well-intentioned researchers to conduct open-label studies leading to highly biased results. In an article published in the
 PLOS journal by three Australian researchers, a review of 960 peer-reviewed articles in 5 different animal behavior journals found that only 6.3% of the studies involved blinded study design. 
Even in clinical research, a sample of 234 meta-analyses of clinical trials showed that about one-third of the meta-analyses did not contain any double-blind studies. In preclinical studies in support of drug approval, of the 2,220 medical experiments on animals published in peer reviewed journals, the data collector was blinded in only 24% of the studies. Not surprisingly, the authors found that in studies that were not blinded, the results were highly exaggerated over blinded studies due to the researchers “expectations” from an experiment clouding their judgment of results. This bias was must stronger in studies that involved subjective variables such as behavioral observations. The trends observed by the authors using text mining and literature review indicated that blinded protocols are still uncommon in the general life sciences research and that nonblind studies tend to report higher effect sizes and more significant p-values. Although the final conclusion of this report is not surprising, the wide-spread use of the non-blinded studies in scientific research is an eye-opener and highlights one of the possible reasons that academic scientific research is often not reproduced in commercial regulated experiments.