ABSTRACT

A test oracle in software testing is a mechanism for checking whether the program under test behaves correctly for any execution. In some practical situations, oracles can be unavailable or too expensive to apply. Metamorphic testing (MT) was proposed to alleviate this problem so that software can be delivered under the time-to-market pressure. However, the effectiveness of MT has not been studied adequately. This paper conducts a controlled experiment to investigate the cost effectiveness of using MT. The fault detection capability and time cost of MT are compared with the standard assertion checking method. Our results show that MT has potentials to detect more faults than the assertion checking method. The experiment results also show a trade-off between the two testing methods: MT can be less efficient but more effective, and can be defined at a coarser level of granularity than the assertion checking method. Keywords: Metamorphic testing, test oracle, controlled experiment, empirical evaluation

1. This research is supported in part by grants of the Research Grants Council of Hong Kong (project numbers 111107 and 717308) and the Australian Research Council (project number DP0984760). 2. Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong. 3. (Corresponding author.) Department of Computer Science, City University of Hong Kong, Tat Chee Avenue, Hong Kong. Tel: +852 2788 9684. Fax: +852 2788 8614. Email: 4. China Merchants Bank, Central, Hong Kong

EVERY VISITOR COUNTS: