Call for Papers
Workshop Description
Linguistic annotation of natural language corpora is the backbone of supervised methods for statistical natural language processing. It also provides valuable data for evaluation of both rule-based and supervised systems and can help formalize and study linguistic phenomena.
The LAW provides a forum for presentation and discussion of innovative research on all aspects of linguistic annotation, including creation/evaluation of annotation schemes, methods for automatic and manual annotation, use and evaluation of annotation software and frameworks, representation of linguistic data and annotations, evaluation of manual and automatically-produced annotations, etc.
Special Theme
We encourage submission of papers relating to this year's theme:
Evaluation of Annotation Quality
This special theme considers current practice in evaluation of linguistic annotations and its successes and failures by asking questions such as: How are we as a community measuring inter-annotator agreement to date, and are there sounder ways to measure it? How can we estimate the annotation quality of existing resources, and what can be done to document annotated data to help others assess its reliability?
We encourage submissions addressing the following topics related to the special theme:
- How agreement is measured in various (new or existing) annotation projects, and what the different scores tell us in each case.
- Good acceptance thresholds for different annotation tasks and metrics, and/or how to determine them.
- Previously proposed but not widely used measures for agreement or annotation quality.
- Proposals for quantitative or qualitative methods to measure agreement or annotation quality.
- Proposals for documentation of published resources to support their evaluation, means and methods to achieve community evaluation of linguistically-annotated resources, etc.
Workshop Topics
We welcome submissions of long and short papers, posters, and demonstrations, relating to the topics of the special theme and to any aspect of linguistic annotation, including:
Annotation procedures
- Innovative automated and manual strategies for annotation.
- Machine learning and knowledge-based methods for automation of corpus annotation.
- Creation, maintenance, and interactive exploration of annotation structures and annotated data.
Annotation evaluation
- Inter-annotator agreement and other evaluation metrics and strategies.
- Qualitative evaluation of linguistic representation.
Annotation access and use
- Representation formats/structures for merged annotations of different phenomena, and means to explore/manipulate them.
- Linguistic considerations for merging annotations of distinct phenomena.
Annotation schemes, guidelines and standards
- New and innovative annotation schemes, comparison of annotation schemes.
- Best practices for annotation procedures and/or development and documentation of annotation schemes.
- Interoperability of annotation formats and/or frameworks among different systems as well as different tasks, frameworks, modalities, and languages.
- Results on the application and evaluation of standards for (linguistic) annotation.
Annotation software and frameworks
- Development, evaluation and/or innovative use of annotation software frameworks.