CheckThat! Lab at CLEF 2023

Home

Editions

Tasks

Contents

Contents

Task 2: Subjectivity in News Articles

Definition

Systems are challenged to distinguish whether a sentence from a news article expresses the subjective view of the author behind it or presents an objective view on the covered topic instead.

This is a binary classification tasks in which systems have to identify whether a text sequence (a sentence or a paragraph) is subjective or objective.

The task is offered in six languages.

Information regarding the annotation guidelines can be found in the following paper: On the Definition of Prescriptive Annotation Guidelines for Language-Agnostic Subjectivity Detection.

Datasets

Evaluation

The official evaluation is macro-averaged F1 between the two classes.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at CheckThat! Lab Task 2 repository

Submission guidelines

  • Make sure that you create one account for each team, and submit it through one account only.
  • The last file submitted to the leaderboard will be considered as the final submission.
  • For subtask 2A, there are 6 languages (Arabic, Dutch, English, German, Italian and Turkish). Moreover, we define a multi-lingual evaluation scenario where we use a balanced sub-sample of all 6 languages to define multi-lingual training and evaluation splits.
  • The name of each output file has to be subtask2A_[LANG}.tsv where LANG can be arabic, dutch, english, german, italian, turkish, or multilingual.
  • Get sure to set .tsv as the file extension; otherwise, you will get an error on the leaderboard.
  • Examples of submission file names should be subtask2A_arabic.tsv, subtask2A_dutch.tsv, subtask2A_english.tsv, subtask2A_german.tsv, subtask2A_italian.tsv, subtask2A_turkish.tsv, subtask2A_multilingual.tsv.
  • You have to zip the tsv into a file with the same name, e.g., subtask2A_arabic.zip, and submit it through the codalab page.
  • If you participate in the task for more than one language, for each language you must do a different submission.
  • For each submission, it is required to submit the team name and method description. Your team name must EXACTLY match the one used during the CLEF registration.
  • You are allowed to submit max 200 submissions per day for each subtask.
  • We will keep the leaderboard private till the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.

The submission is done through the Codalab platform at https://codalab.lisn.upsaclay.fr/competitions/13046

Leaderboard

Multilingual

Team Macro F1 SUBJ F1
1 NN 0.82 0.81
- tarrekko 0.81 0.81
2 + Thesis Titan 0.81 0.81
- ES-VRAI 0.78 0.77
3 BASELINE 0.74 0.75
4 TOBB ETU 0.67 0.73

* Submissions without position were submitted after the deadline.

* Submissions with a + sign include task organisers

Arabic

Team Macro F1 SUBJ F1
1 NN 0.79 0.67
- tarrekko 0.79 0.67
2 Thesis Titan 0.78 0.64
3 Accenture 0.73 0.58
4 BASELINE 0.66 0.50
5 TOBB ETU 0.65 0.52

* Submissions without position were submitted after the deadline.

Dutch

Team Macro F1 SUBJ F1
1 + Thesis Titan 0.81 0.80
- tarrekko 0.78 0.75
2 NN 0.76 0.71
3 TOBB ETU 0.73 0.77
4 BASELINE 0.67 0.65
5 Accenture 0.62 0.72

* Submissions without position were submitted after the deadline.

* Submissions with a + sign include task organisers

English

Team Macro F1 SUBJ F1
- tarrekko 0.78 0.78
1 DWReCo 0.78 0.78
2 gpachov 0.77 0.77
3 Thesis Titan 0.77 0.79
4 KUCST 0.73 0.71
5 NN 0.73 0.73
6 Fraunhofer SIT 0.73 0.77
7 BASELINE 0.72 0.73
8 Accenture 0.69 0.64
9 TOBB ETU 0.63 0.74
10 Awakened 0.60 0.59
11 TUDublin 0.40 0.24

* Submissions without position were submitted after the deadline.

German

Team Macro F1 SUBJ F1
1 Thesis Titan 0.82 0.77
2 NN 0.74 0.67
- tarrekko 0.73 0.68
3 TOBB ETU 0.71 0.67
4 DWReCo 0.70 0.64
5 Fraunhofer SIT 0.68 0.65
6 BASELINE 0.64 0.57
7 Accenture 0.26 0.50

* Submissions without position were submitted after the deadline.

Italian

Team Macro F1 SUBJ F1
1 Thesis Titan 0.76 0.65
- tarrekko 0.72 0.60
2 NN 0.71 0.58
3 Accenture 0.66 0.55
4 BASELINE 0.64 0.52
5 TOBB ETU 0.63 0.54
6 TUDublin 0.46 0.11

* Submissions without position were submitted after the deadline.

Turkish

Team Macro F1 SUBJ F1
1 Thesis Titan 0.90 0.91
- tarrekko 0.87 0.88
2 DWReCo 0.84 0.85
3 NN 0.81 0.80
4 Accenture 0.78 0.82
5 BASELINE 0.77 0.79
6 TOBB ETU 0.70 0.79

* Submissions without position were submitted after the deadline.

Organizers

  • Andrea Galassi, Università di Bologna, Italy
  • Federico Ruggeri, Università di Bologna, Italy
  • Alberto Barrón-Cedeño, Università di Bologna, Italy

Arabic data

  • Firoj Alam, Qatar Computing Research Institute, HBKU, Qatar
  • Maram Hasanain, Qatar Computing Research Institute, HBKU, Qatar
  • Wajdi Zaghouani, HBKU, Qatar

Dutch data

  • Tommaso Caselli, University of Groningen, The Netherlands
  • Folkert Leistra, University of Groningen, The Netherlands

English data

  • Francesco Antici, Università di Bologna, Italy
  • Alessandra Bardi, Università di Bologna, Italy
  • Alice Fedotova, Università di Bologna, Italy
  • Katerina Korre, Università di Bologna, Italy
  • Arianna Muti, Università di Bologna, Italy

German data

  • Julia Maria Struß, University of Applied Sciences Potsdam, Germany
  • Juliane Köhler, University of Applied Sciences Potsdam, Germany
  • Melanie Siegel, Darmstadt University of Applied Sciences, Germany
  • Michael Wiegand, University of Klagenfurt, Austria
  • Katja Ebermanns, University of Applied Sciences Potsdam, Germany

Italian data

  • Francesco Antici, Università di Bologna, Italy
  • Alessandra Bardi, Università di Bologna, Italy
  • Alice Fedotova, Università di Bologna, Italy
  • Arianna Muti, Università di Bologna, Italy

Note: The Italian training and validation dataset is partially derived from SubjectivITA corpus.

Turkish data

  • Mucahid Kutlu, TOBB University of Economics and Technology, Ankara, Turkey
  • Mehmet Deniz Turkmen, TOBB University of Economics and Technology, Ankara, Turkey

Contact

For queries, please join the Slack channel

You can access it directly at https://ct-23-participants.slack.com

Alternatively, please send an email to: clef-factcheck@googlegroups.com