Transformer-Based Automatic Item Generation for Course-Based Test Items: A Case Study of Translation Tasks in China’s Context

Abstract

In order to meet the rapidly increasing demand for item pools for large-scale assessments, automatic item generation (AIG) emerged about thirty years ago, using pre-programmed algorithms to automatically construct large numbers of test items with predictable item parameters. The rapid progress in natural language processing due to transformer networks has enabled large language models to handle a variety of natural language processing tasks, i.e., translation, text summarization, question answering, and writing text, at a level similar to humans. This study has carried out an empirical study on the human and transformer-based collaborative AIG framework for course-based item generation performances of several GenAI models for translation tasks of the English examination in China. The results show that: 1) Most GenAI models can successfully generate English-Chinese and Chinese-English sentence translation items. 2) Most GenAI models can generate both English-Chinese and Chinese-English text translation passages. 3) Readability of the generated passages is analyzed, and content validity of the generated sentence translation items and text translation passages is verified by subject matter experts. This study highlights that GenAI models help reduce teachers’ burdens of repetitive and time-consuming human item writing tasks if handled properly.