Gender Issues between Gemini and ChatGPT: The Case of English-Arabic Translation

Abstract

The study focuses on the gender-related issues that face English-Arabic machine translation. It aims to investigate and evaluate gender accuracy in the translations provided by two prominent large language models, Gemini and ChatGPT, recognizing the rich morphological system of Arabic that includes gender marking. The researchers develops a test suite to evaluate gender accuracy in the translation outputs of Gemini and ChatGPT. The evaluation is performed by two professional annotators. That is followed by an analysis of the patterns of the gender-related issues that appear in the translation outputs of  the models under study. The results show that Gemini outperformed ChatGPT in almost every aspect when it comes to gender-related translation issues. Both the number of the annotated issues as well as the gender accuracy evaluation came in favor of Gemini. The study introduced different patterns of gender-related translation issues. It also provides recommendations for future research.