IGK 2004 : Project Proposal

Towards an Automated Mixed-Media Answer Summary System
A Next-Generation Question Answering System

Links:

- Wee/QAAM man page
- NEW!! Slides (Project Presentation)
- Directory Sample here

Runs: (local access only)

- run 1

Description

In work to date on Question Answering, I have shown that even simple 'factoid' questions often have more than one answer. As part of this work, I have built a small web-based QA system called Wee, which organises the result of queries sent to a search engine and provides answers such as those shown in Table 1, in response to the question "What is Wimbledon?".

However, while such answers are more informative than a list of documents, they are flat structures and the connection between their members is still not clear: Are answers alternatives or aggregations or a mix of both? (In Table 1, answers 2 and 5 are clearly related whereas 1 is a different club, 3 and 4 provide more general information) .

Our proposal is to set up a QA renderer that would make explicit the underlying relations between answers for a better and faster user interpretation. A QA renderer should be different from existing IR renderers [Honkela:1997:phd, Cugini:1996:cikm], which are mainly topic maps. QA is more accurate and requires an answer organisation based on word senses or referents rather than topics.

The proposed project involves the development of a mixed-media answer summary system. The most salient senses of the answer will be presented with a short summary, a relevant image of the answer entity or its context, a set of links to other relevant pictures and a set of links providing references for the answer. A possible example of such a rendering is given in Figure 1.

Summaries will organize information in a more compact and structured way than a simple list of sentences and prevent the user from "jumping" to one line to the other to check if the answers are actually the same answers or not. Pictures will provide a straight understanding of the textual answer either by a direct visualisation or by a new form of context. Links will redirect the user to more information.

This project would be benefit of an existing framework consisting in a web QA system (Wee) and a post-processor called QAAM.
QAAM (for QA Answer Model, [Dalmas:2004:cluk]) fuses and organizes information into a model before passing it on to a renderer. This framework distinguishes between data and rendering, making the QA process compliant with the Model-View-Control design pattern [Krasner:1988] that has proved to be a successful engineering methodology to implement user interfaces.
QAAM produces a graph of answers currently not exploited by the list rendering. The objective of this project is to implement a mixed-media renderer for QAAM (This project does NOT consist in finding answers).

Figure 1: Answer Rendering WIMBLEDON a private Club founded in 1868 , originally as ' The All England Croquet Club ' and its first ground was situated off Worple Road oldest and most prestigious event in the sport of tennis More picture(s) [1] [2] Reference(s) http://explanation-guide.info/ http://www.wimbledon.org/en Wimbledon Football Club Limited More picture(s) [1] Reference(s) http://www.courtservice.gov.uk/judgmentsfiles/ area in south-west London More picture(s) None Reference(s) http://www.wimbledonhomesb-b.co.uk/locality.htm thought to have come from the saxon word ' Ton ' meaning ' enclosed ' or ' Don ' meaning ' hill ' More picture(s) None Reference(s) http://www.wordiq.com/definition/Wimbledon

Implementation

We will use existing web answers found by Wee to a subset of TREC 10 questions. Given the timeline, we will focus on two or three specific question types only (see data set below).

We propose to implement a renderer based on three sub-components:

- a summarizer that would pick up and filter sentences related to one answer sense (provided by QAAM) and organize them. Summarization will help removing redundant answers and organize together answers aggregated by QAAM.

- an image filter using Google images to retrieve relevant pictures. The query will be based on information provided by QAAM (question words and keywords defining an answer cluster). Relevant pictures will have to be filtered using heuristics (URL, image name).

- a presentation mode selection module dependent on question type

Challenges

This project is meant to be a proof of concept. A fully automated QA renderer would be a step towards "next-generation" QA systems as described by [Burger:2002] and open the door to new research such as:
- how to evaluate complex answers
- how to evaluate generated answers
- how to evaluate multimedia answers

It would be a direct competitor to the MIT START system [START], which provides mixed-media but based on partial human annotation and filtering.

Data set

Question types:

location (58 questions)
Where is Perth?
What is the capital of Mongolia?

definition (108 questions)
What is nicotine?
What are invertebrates?

famous_for (3 questions)
Who was Galileo?
Who is Duke Ellington?

Answers to be used (correct/incorrect)

Number of correct answers per question (multiple answers)

References

START page: http://www.ai.mit.edu/projects/infolab/
Publications: http://www.ai.mit.edu/projects/infolab/publications/

NIRVE: http://zing.ncsl.nist.gov/~cugini/uicd/nirve-home.html
[Cugini:1996:cikm] J. Cugini and C. Piatko and S. Laskowski, Interactive 3D Visualization for Document Retrieval, Workshop on New Paradigms in Information Visualization and Man ipulation , ACM Conference on Information and Knowledge Management, 1996

Self-Organization Maps for IR WEBSOM: http://websom.hut.fi/websom/
[Honkela:1997:phd] Timo Honkela, Self-Organizing Maps in Natural Language Processing, Ph.D., Helsinki University of Technology, 1997

[Andree:2000] Andree, Elisabeth: The Generation of Multimedia Documents, in: R. Dale, H. Moisl and H. Somers: A Handbook of Natural Language Processing: Techniques and Applications for the Processing of Language as Text, Marcel Dekker Inc., 305-327, 2000

[Krasner:1988] G. Krasner and S. Pope, A Cookbook for Using the Model-View-Controller User Interface Paradigm in Smalltalk-80, Journal of Object-Oriented Programming (JOOP), 1988

[Barzilay:1999:acl] R. Barzilay and K. R. McKeown and M. Elhadad, Information Fusion in the Context of Multi-Document Summarization, 37th Annual Meeting of the Association for Computational Linguistics, 550-557, 1999

[Burger:2002] J. Burger and C. Cardie and V. Chaudhri and R. Gaizauskas and S. Harabagiu and D. Israel and C. Jacquemin and C. Lin and S. Maiorano and G. Miller and D. Moldovan and B. Ogden and J. Prager and E. Riloff and A. Singhal and R. Shrihari and T. Strzalkowski and E. Voorhees and R. Weishedel, Issues, Tasks and Program Structures to Roadmap Research in Question and Answering, NIST, 2002.

[Dalmas:2004:cluk] T. Dalmas and B. Webber, Answer Comparison: Analysis of Relationships between Answers to `Where'-Questions, 7th Computational Linguistics UK (CLUK), 2004