Task 1: Subjectivity
Definition
Systems are challenged to distinguish whether a sentence from a news article expresses the subjective view of the author behind it or presents an objective view on the covered topic instead.
This is a binary classification tasks in which systems have to identify whether a text sequence (a sentence or a paragraph) is subjective (SUBJ) or objective (OBJ).
The task comprises three settings:
- Monolingual: train and test on data in a given language L
- Multilingual: train and test on data comprising several languages
- Zero-shot: train on several languages and test on unseen languages
Datasets
We provide training data in five languages:
Note: Test datasets will be made available once the evaluation cycle starts.
Note: For multilingual and zero-shot settings, all language-specific training data can be used.
Annotation Guidelines
Information regarding the annotation guidelines can be found in the following paper: A Corpus for Sentence-Level Subjectivity Detection on English News Articles.
Evaluation
The official evaluation is macro-averaged F1 between the two classes.
Organizers
- Federico Ruggeri, DISI, University of Bologna, Italy,
- Arianna Muti, MilaNLP, Bocconi, Milan
- Katerina Korre, DIT, University of Bologna, Italy
Note: Language-specific curators will be announced once the evaluation cycle ended.
For queries, please join the Slack channel
Alternatively, please send an email to: clef-factcheck@googlegroups.com