Task 4: Factuality of Reporting of News Media
Definition
In this task we ask to identify the factuality of reporting at the media level. Given the news article(s) of a news outlet (e.g., www.cnn.com), the task asks to predict the factuality of reporting of that news outlet: low, mixed, and high. We offer the task in English
Evaluation
This is an ordinal classification task, and we will use mean absolute error as the official measure.
Submission
All scripts can be found on gitlab at CheckThat! Lab Task 4 repository
Submission guidelines
- Make sure that you create one account single account for your team, and submit runs through that account only.
- The last file submitted to the leaderboard will be considered as the final submission.
- The file with your predictions should be called
task4.tsv
. Get sure to set .tsv
as the file extension; otherwise, you will get an error on the leaderboard. Subtasks are 1A and 1B. For subtask 1A, there are two languages (Arabic, and English). For subtask 1B, there are three languages (Arabic, Spanish, and English). For instance, a submission file for task 1B Arabic should be subtask1B_arabic.tsv
.
- You have to zip the tsv, ‘zip task4.zip task4.tsv’ and submit it through the codalab page.
- You have to include the team name and the description of your method each submission. Your team name must EXACTLY match the one used during the CLEF registration.
- You are allowed to submit max 200 submissions per day for each subtask.
- We will keep the leaderboard private until the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.
Submission Site
Task 4: Codalab
Leaderboard
The baseline is a random system.
|
Team |
MAE |
1 |
CUCPLUS |
0.295 |
2 |
NLPIR-UNED |
0.344 |
3 |
Accenture |
0.467 |
4 |
UBCS |
0.541 |
5 |
Awakened |
0.705 |
6 |
Random |
0.943 |
Organizers
- Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence
- Firoj Alam, Qatar Computing
Research Institute, HBKU
- Giovanni Da San Martino, University of Padova
- Maram Hasanain, Qatar Computing Research Institute, HBKU
- Dilshod Azizov, Mohamed bin Zayed University of Artificial Intelligence
- Rabindra Nath Nandi, BJIT Limited
- Panayot Panayotov, Sofia University
Please join on Slack channel for any queries:
https://join.slack.com/t/newworkspace-qgd1635/shared_invite/zt-1k9lnm0ys-aMeEmUtGY0tfW5HbkXHPJg
Slack Channel: https://ct-23-participants.slack.com
Or send an email to: clef-factcheck@googlegroups.com