Task 4: Factuality of Reporting of News Media

Definition

In this task we ask to identify the factuality of reporting at the media level. Given the news article(s) of a news outlet (e.g., www.cnn.com), the task asks to predict the factuality of reporting of that news outlet: low, mixed, and high. We offer the task in English

Datasets

Task 4 dataset

Evaluation

This is an ordinal classification task, and we will use mean absolute error as the official measure.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at CheckThat! Lab Task 4 repository

Submission guidelines

Make sure that you create one account single account for your team, and submit runs through that account only.
The last file submitted to the leaderboard will be considered as the final submission.
The file with your predictions should be called task4.tsv. Get sure to set .tsv as the file extension; otherwise, you will get an error on the leaderboard. Subtasks are 1A and 1B. For subtask 1A, there are two languages (Arabic, and English). For subtask 1B, there are three languages (Arabic, Spanish, and English). For instance, a submission file for task 1B Arabic should be subtask1B_arabic.tsv.
You have to zip the tsv, ‘zip task4.zip task4.tsv’ and submit it through the codalab page.
You have to include the team name and the description of your method each submission. Your team name must EXACTLY match the one used during the CLEF registration.
You are allowed to submit max 200 submissions per day for each subtask.
We will keep the leaderboard private until the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.