IEEE/ACM 5th International Workshop on Metamorphic Testing (MET '20), Proceedings of the IEEE/ACM 42nd
International Conference on Software Engineering Workshops
(ICSEW '20), ACM, New York, NY, pp. 388-395 (2020)
Metamorphic Robustness Testing of Google Translate 1

Dickson T.S. Lee 2 , Zhi Quan Zhou 3 , and T.H. Tse 2

[author-izer free download from ACM digital library]

 ABSTRACT

Current research on the testing of machine translation software mainly focuses on functional correctness for valid, well-formed inputs. By contrast, robustness testing, which involves the ability of the software to handle erroneous or unanticipated inputs, is often overlooked. In this paper, we propose to address this important shortcoming. Using the metamorphic robustness testing approach, we compare the translations of original inputs with those of followup inputs having different categories of minor typos. Our empirical results reveal a lack of robustness in Google Translate, thereby opening a new research direction for the quality assurance of neural machine translators.

Keywords: Robustness testing, Oracle problem, Metamorphic testing, Metamorphic robustness testing, Machine translation, MT4MT

1. This work was supported in part by a linkage grant of the Australian Research Council (Project ID: LP160101691) and a Western River entrepreneurship grant.
2. Department of Computer Science, The University of Hong Kong, Pokfulam, Hong Kong.
3. (Corresponding author.)
School of Computer Science and Software Engineering, University of Wollongong, Wollongong, NSW 2522, Australia.
Email:

 EVERY VISITOR COUNTS:

  Cumulative visitor count