SemEval 2010 Task:

Linking Events and their Participants in Discourse

This page provides further information on the task, specifically on:

Research Areas Related to the Task
Background Information
Sub-Tasks
Data
Time Schedule, Release Dates
Organising Team
Contact

For more information on the available data sets click on Downloads. For detailed information on the formats click on Info for Participants.

Research Areas

This shared task should appeal to researchers working on:

semantic role labelling (FrameNet or PropBank)
co-reference resolution, or
information extraction

Semantic role labelling (SRL) has traditionally been viewed as a sentence-internal problem. However, it is clear that there is an interplay between local semantic argument structure and the surrounding discourse. In this shared task, we would like to take SRL of nominal and verbal predicates beyond the domain of isolated sentences by linking local semantic argument structures to the wider discourse context. In particular, we aim to find fillers for roles which are left unfilled in the local context (null instantiations, NIs). An example is given below, where the "charges" role ("arg2" in PropBank) of cleared is left empty but can be linked to murder in the previous sentence.

In a lengthy court case the defendant was tried for murder. In the end, he was cleared.

Sub-Tasks

There will be two tasks, which will be evaluated independently (participants can decide to enter either or both):

For the Full Task the target predicates in the (test) data set will be annotated with gold standard word senses (i.e., frames). The participants have to:

find the semantic arguments of the predicate (role recognition) and label them with the correct role (role labelling)
find links between null instantiations and the wider context (NI linking)

For the NIs only task, participants will be supplied with a test set which is already annotated with gold standard local semantic argument structure (frames and roles); only the referents for null instantiations have to be found.

For more details, also on the evaluation, see the following paper:

Josef Ruppenhofer, Caroline Sporleder, Roser Morante, Collin Baker and Martha Palmer. SemEval-2010 Task 10: Linking Events and Their Participants in Discourse. The NAACL-HLT 2009 Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-09), Boulder, Colorado, USA, June 4, 2009.

Data

We annotate data of running text from the fiction domain. The training set is available here. The test set will be made available later this year. The data are freely available. The training set for both tasks will be annotated with gold standard semantic argument structure (see for example the FrameNet full text annotation) and linking information for null instantiations. We annotate the semantic argument structures both in FrameNet and PropBank style; participants can choose which one they prefer. Since the FrameNet and PropBank annotations are not entirely comparable, systems which work with the former will be evaluated separately from those that use the latter.

Time Schedule

Trial Data Release: 16 September 2009
Training Data Release: 25 January 2010
Test Data Release: 26 March 2010
Results due: 2 April 2010 (Note: this is a hard deadline for all SemEval 2010 tasks)

Organising Team

Josef Ruppenhofer (Saarland University)
Caroline Sporleder (Saarland University)
Roser Morante (University of Antwerp)
Collin Baker (ICSI, Berkeley)
Martha Palmer (University of Colorado, Boulder)

Contact

We have created a Google group for the participants of Task-10.

The organizers are also on the list and will follow the discussion there. If you want to contact us about something off the list, you can write to us directly. For now, the main contact will be: @coli.uni-sb.de