A publication bias arises when editors and reviewers exhibit a preference for publishing statistically significant results in contrast with methodologically sound studies reporting nonsignificant results. To test whether such a bias exists, Atkinson et al. (1982) submitted bogus manuscripts to 101 consulting editors of APA journals. The submitted manuscripts were identical in every respect except that some results were statistically significant and others were nonsignificant. Editors received only one version of the manuscript and were asked to rate the manuscripts in terms of their suitability for publication. Atkinson et al. found that manuscripts reporting statistically nonsignificant findings were three times more likely to be recommended for rejection than manuscripts reporting statistically significant results. A similar conclusion was reached by Coursol and Wagner (1986) in their survey of APA members. These authors found that 80% of submitted manuscripts reporting positive outcome studies were accepted for publication in contrast with a 50% acceptance rate for neutral or negative outcome studies.

The existence of a publication bias is a logical consequence of null hypothesis significance testing. Under this model the ability to draw conclusions is essentially determined by the results of statistical tests. As we saw in Chapter 3, the shortcoming of this approach is that p values say as much about the size of a sample as they do about the size of an effect. This means that important results are sometimes missed because samples were too small. A nonsignificant result is an inconclusive result. A nonsignificant p tells us that there is either no effect or there is an effect but we missed it because of insufficient power. Given this uncertainty it is not unreasonable for editors and reviewers to exhibit a preference for statistically significant conclusions.1 Neither should we be surprised that researchers are reluctant to write up and report the results of those tests that do not bear fruit. Not only will they find it difficult to draw a conclusion (leading to the awful temptation to do a post hoc power analysis), but the odds of getting their result published are stacked against them. Combine these two perfectly rational tendencies - selective reporting and selective publication - and you end up with a substantial availability bias.