This project aims to develop a toolkit that supports evaluation of writings in Japanese as a second language by integrating implicit expertise of human assessment and computational algorithms based on machine learning techniques.
To make this possible, we first collect Japanese writings by learners in eleven countries (Croatia, France, Germany, Hungary, Italy, Netherlands, Serbia, Slovenia, Spain, Russia, United States of America) to build a corpus of text by learners in Western countries, which has not necessarily been easy to obtain before. The dataset comprises the responses to five types of prompts that require the participants to structure text in different rhetorical modes, including narrative, descriptive, comparative expository, and argumentative. Next, we manually evaluate those responses and assign proficiency levels to each, then apply computational analysis in order to extract statistical characteristic quantities of the text. Utilizing the results from these manual and computational processes, finally, we develop an automated evaluation system that analyzes given input text and yields a level of accuracy close to that of experienced human evaluators.
The ability of writing in one’s second language consists of the skills to structure a coherent organization of text and content on the one hand (i.e., writing expertise) and the more basic skills regarding composition of grammatical sentences and expressions (i.e., L2 proficiency). It is basically the latter type of proficiency that can be assessed using automated computational processes. Thus, our primary focus will be placed on this rather basic aspect to a learner’s writing skill. The online evaluation system that we develop, however, will implement functionality to show a rough sketch of text organization by visually highlighting “macro structure” and “meta-language” of the input data.
In the final year of the research period, we will make the online evaluation system available to the public. In addition, samples of writing corresponding to different levels of proficiency will be released. It is expected that the whole toolkit will be of significant assistance to educators worldwide in the practice of writing in Japanese.