SemEval 2010 Task:
Linking Events and their Participants in Discourse
This page provides further information on the task, specifically on:
- Research Areas Related to the Task
- Background Information
- Time Schedule, Release Dates
- Organising Team
- semantic role labelling (FrameNet or PropBank)
- co-reference resolution, or
- information extraction
Semantic role labelling (SRL) has traditionally been viewed as a
sentence-internal problem. However, it is clear that there is an
interplay between local semantic argument structure and the
surrounding discourse. In this shared task, we would like to take SRL
of nominal and verbal predicates beyond the domain of isolated
sentences by linking local semantic argument structures to the wider
discourse context. In particular, we aim to find fillers for roles
which are left unfilled in the local context (null instantiations,
NIs). An example is given below, where the "charges" role ("arg2" in
PropBank) of cleared is left empty but can be
murder in the previous sentence.
In a lengthy court case the defendant was tried for murder. In the end, he was cleared.
There will be two tasks, which will be evaluated independently (participants can decide to enter either or both):
For the Full Task the target predicates in the (test) data set will be annotated with gold standard word senses (i.e., frames). The participants have to:
- find the semantic arguments of the predicate (role recognition) and label them with the correct role (role labelling)
- find links between null instantiations and the wider context (NI linking)
For the NIs only task, participants will be supplied with a test set which is already annotated with gold standard local semantic argument structure (frames and roles); only the referents for null instantiations have to be found.
For more details, also on the evaluation, see the following paper:
Josef Ruppenhofer, Caroline Sporleder, Roser Morante, Collin Baker and Martha Palmer. SemEval-2010 Task 10: Linking Events and Their Participants in Discourse. The NAACL-HLT 2009 Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-09), Boulder, Colorado, USA, June 4, 2009.
We annotate data of running text from the fiction domain. The training set is available here. The test set will be made available later this year. The data are freely available. The training set for both tasks will be annotated with gold standard semantic argument structure (see for example the FrameNet full text annotation) and linking information for null instantiations. We annotate the semantic argument structures both in FrameNet and PropBank style; participants can choose which one they prefer. Since the FrameNet and PropBank annotations are not entirely comparable, systems which work with the former will be evaluated separately from those that use the latter.
- Trial Data Release: 16 September 2009
- Training Data Release: 25 January 2010
- Test Data Release: 26 March 2010
- Results due: 2 April 2010 (Note: this is a hard deadline for all SemEval 2010 tasks)
- Josef Ruppenhofer (Saarland University)
- Caroline Sporleder (Saarland University)
- Roser Morante (University of Antwerp)
- Collin Baker (ICSI, Berkeley)
- Martha Palmer (University of Colorado, Boulder)
We have created a Google
group for the participants of Task-10.
The organizers are also on the list and will follow the discussion there. If you want to contact us about something off the list, you can write to us directly. For now, the main contact will be: @coli.uni-sb.de