Analyses of factors related to positive test bias in software testing

In earlier work, we have shown that software testers exhibit positive test bias. Positive test bias is the pervasive behavioral phenomenon in which hypothesis testers tend to test a hypothesis with data which confirms the hypothesis. However, in software testing this behavior may be counter-productive, since it may be more effective to test with data which are designed to disconfirm the hypothesis. The first study considered how positive test bias is influenced by the expertise level of the subjects, the completeness of the software specifications and whether or not the programs contained errors. The results demonstrated strong evidence of positive test bias regardless of condition. The effects appear to be partially mitigated by increasingly higher levels of expertise and by increasingly more complete specifications. In some cases, the effect is also increased by the presence of errors. A second study used talk-aloud protocols to explore the kinds of hypotheses testers generate during testing. The results further emphasize that subjects test their programs in a biased way and support the notion that the program specification drives testers’ hypotheses. We conclude that positive test bias is a critical concern in software testing and may have a seriously detrimental effect on the quality of testing. The results further emphasize the importance of complete and thorough program specifications in order to enhance effective testing.