Bootstrapping a Reinforcement Learning-based System for Learning Clarification Strategies
Speaker: Verena Rieser
Institution: Saarland University
Abstract:
In this talk I report on our recent progress in bootstrapping a Reinforcement Learing (RL)-based strategy for Learning Clarification Strategies for multimodal dialogue systems. After giving a short overview of the overall framework and the data, I discuss two aspects of my work in more detail.
First, I present a novel cluster-based technique for building user simulations for learning dialogue strategies, which show varying, but complete and consistent behaviour with respect to real users [Rieser and Lemon, Interspeech/ICSLP 2006]. We use this technique to build user simulations and we also introduce the SUPER (Simulated User Pragmatic Error Rate) evaluation metric which allows us to evaluate user simulations with respect to these desiderata. We show that the cluster-based user simulation technique performs significantly better (at P<0.01) than decisions made using either the one most likely action or a random baseline. The cluster-based user simulations reduce the average error of these other models by 53% and 34% respectively.
If time allows, I will also report on current work on using Logistic Regression to initialise RL-based Dialogue Systems from data gathered Wizard-of-Oz studies. We use these models to initialise the state space for RL with features which influence a human wizard strategy. Furthermore we show that for predicting clarification behaviour the logistic models improve over the baseline on average twice as much as the supervised learning techniques used in previous work [Rieser and Lemon, ACL/COLING 2006], [Williams and Young, SigDial 2003]. In future work we will investigate the benefits of good initial policies for learning dialogue strategies.