Computational Linguistics Colloquium

Thursday, 20 November, 16:15
Conference Room, Building C7 4

Diagnosing meaning errors in Intelligent Computer-Assisted Language Learning

To integrate intelligent computer-aided language learning (ICALL) systems into foreign language teaching practice, such systems arguably need to offer meaning-based, contextualized activities. Current ICALL systems can detect and provide feedback for incorrect grammatical forms used by a learner. But in most (I)CALL systems, the appropriateness of answer content is simply determined by string matching: if the input string matches that of a stored target answer, then the input answer is interpreted as correct. Clearly matching meaning through form in this way severely restricts the types of exercises that can be offered to learners given that the expected input variation must be tightly controlled. At the other end of the spectrum, some projects have advocated supporting learner input that is entirely unconstrained in contents and form, requiring extensive robust form and content analysis as part of a very ambitious research agenda.

In this joint work with Stacey Bailey, we want to argue that it useful to explore the spectrum between the extremes, to determine what level of content analysis is appropriate for which kind of contextualized activities. An interesting type of activity in the middle ground are short-answer reading comprehension questions for which learner answers may vary widely in form, but there is an expected meaning which can be expressed in a target response. The task of comparing the meaning of the target and the learner answer can be seen as related to current CL tasks such as paraphrase or textual entailment detection. But different from those it is a contextualized task: the meaning of the target and the learner answers are compared in relation to the reading comprehension question and the text which this question is about, which avoids some of the problems that have been pointed out for general, uncontextualized textual entailment. The Content Assessment Module we have developed for English reaches an accuracy of 88% for semantic error detection and 87% for a more fine-grained semantic error diagnosis on previously unseen test data drawn from real-life foreign language classrooms.

If you would like to meet with the speaker, please contact Magdalena Wolska.