Computational Linguistics & Phonetics Computational Linguistics & Phonetics Fachrichtung 4.7 Universität des Saarlandes

Generating instructions in virtual environments

Softwareprojekt for BSc and MSc students (Coli)
CS students welcome (email me)
Wintersemester 2009/10
Alexander Koller

Time and place: Thu 10-12, "Aquarium" meeting room, C7.4

The generation of natural-language instructions that help a user perform some task in a virtual environment is a hip new task in the field of natural language generation (NLG). This is primarily due to the Challenge for Generating Instructions in Virtual Environments (GIVE). In this challenge, the NLG system has access to complete symbolic information about the world, and can react to the user's actions in the world in real time. The challenge is interesting because it offers a new answer to the old problem of evaluating NLG systems, namely by connecting NLG systems to human experimental subjects over the Internet; in GIVE-1, we collected data from almost 1200 such subjects. It is also interesting because it moves the focus in NLG away from the generation of pure text, towards the much more realistic and challenging setting of generation in the context of a spatial environment, of real-time responses to events, and of an overarching task that has to be solved.

In this software project, the students will form up in teams and then design and develop NLG systems that we will then submit to the GIVE-2 Challenge, which has just started. Thus, this seminar offers you the chance to come up with creative ideas for instruction generation, implement them in a practical system that will be used by several hundred people over the Internet, and compete with other research teams around the world: So far, ten teams from seven countries have signed up. We will start the course with an overview of NLG and of previous approaches to GIVE, and talk about programming in teams. Then it will be up to you (with my help) to build a GIVE system.

The GIVE software is implemented in Java. This means that you should be comfortable with Java or some other language that compiles to Java bytecode, or be willing to invest the extra effort to become comfortable with it during the course. If you want to try out GIVE-2 with a prototype NLG system, you can do that here. If you want to get an idea of what's involved in implementing a GIVE-2 system, you can download the software and read some documentation here. To have a look at what the systems in GIVE-1 were like, see here. If you have any further questions, please feel free to send me an email.