Practical
Methods for Answer Comparison in Question Answering: Motivations,
Techniques and Evaluation
Tiphaine Dalmas
Keywords: Question Answering,
Information fusion, Model-View-Controller design pattern
Multiple answers are a frequent
occurrence in automated Question Answering (QA). However, current
evaluation methods (TREC, CLEF) do not give them the consideration they
seriously deserve: Redundancy is penalized, but systems are not
required to recognize that different answers are possible or to do
anything sensible to present them.
We describe a system, QAAM, which generates an answer model from
extractions provided by our web-based QA system. An answer model is an
oriented graph in which nodes are extractions and edges represent
lexical and contextual relations automatically discovered by QAAM.
We have evaluated QAAM on TREC 10 and 11 and report two kinds of
improvement. Quantitatively, more answers are found when information is
fused into a model. Qualitatively, a model provides meta-information
that allows the system to distinguish among different answer topics and
helps organize and generate the final output (detailed/short answer,
summary/picture/computer readable format), depending on the
end user's requirements. The latter forms the basis of the
project we are carrying out at the current IGK Summer School.