CheckThat! Lab at CLEF 2023

Home

Editions

Tasks

Contents

Task 4: Factuality of Reporting of News Media

Definition

In this task we ask to identify the factuality of reporting at the media level. Given the news article(s) of a news outlet (e.g., www.cnn.com), the task asks to predict the factuality of reporting of that news outlet: low, mixed, and high. We offer the task in English

Datasets

Task 4 dataset

Evaluation

This is an ordinal classification task, and we will use mean absolute error as the official measure.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at CheckThat! Lab Task 4 repository

Submission guidelines

  • Make sure that you create one account single account for your team, and submit runs through that account only.
  • The last file submitted to the leaderboard will be considered as the final submission.
  • The file with your predictions should be called task4.tsv. Get sure to set .tsv as the file extension; otherwise, you will get an error on the leaderboard. Subtasks are 1A and 1B. For subtask 1A, there are two languages (Arabic, and English). For subtask 1B, there are three languages (Arabic, Spanish, and English). For instance, a submission file for task 1B Arabic should be subtask1B_arabic.tsv.
  • You have to zip the tsv, ‘zip task4.zip task4.tsv’ and submit it through the codalab page.
  • You have to include the team name and the description of your method each submission. Your team name must EXACTLY match the one used during the CLEF registration.
  • You are allowed to submit max 200 submissions per day for each subtask.
  • We will keep the leaderboard private until the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.

Submission Site

Task 4: Codalab

Leaderboard

The baseline is a random system.

Team MAE
1 CUCPLUS 0.295
2 NLPIR-UNED 0.344
3 Accenture 0.467
4 UBCS 0.541
5 Awakened 0.705
6 Random 0.943

Organizers

  • Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence
  • Firoj Alam, Qatar Computing Research Institute, HBKU
  • Giovanni Da San Martino, University of Padova
  • Maram Hasanain, Qatar Computing Research Institute, HBKU
  • Dilshod Azizov, Mohamed bin Zayed University of Artificial Intelligence
  • Rabindra Nath Nandi, BJIT Limited
  • Panayot Panayotov, Sofia University

Contact

Please join on Slack channel for any queries: https://join.slack.com/t/newworkspace-qgd1635/shared_invite/zt-1k9lnm0ys-aMeEmUtGY0tfW5HbkXHPJg

Slack Channel: https://ct-23-participants.slack.com

Or send an email to: clef-factcheck@googlegroups.com