Definition

The aim of this task is to determine whether a claim in a tweet is worth fact-checking. Typical approaches to make that decision require to either resort to the judgments of professional fact-checkers or to human annotators to answer several auxiliary questions such as “does it contain a verifiable factual claim?”, and “is it harmful?”, before deciding on the final check-worthiness label.

This year we offer two kinds of data, which translate to two subtasks:

• Subtask 1A (Multimodal): Tweets includes both a text snippet and an image have to be assessed for check-worthiness.
• Subtask 1B (Multigenre): A text snippet alone —from a tweet or a debate/speech transcription— has to be assessed for check-worthiness.

Subtask 1A is offered in Arabic and English, whereas Subtask 1B is offered in Arabic, English and Spanish.

Datasets

Subtask 1A (Multimodal):

• Arabic
• English

Each instance is composed ot the text and the image associated to a tweet.

Subtask 1B (Multigenre):

• Arabic
• English
• Spanish

Each instance is composed of only text, which could come from a tweet, the transcription of a debate or the transcription of speech.

Evaluation

This is a binary classification task. The official evaluation metric is F_1 over the positive class.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at CheckThat! Lab Task 1 repository

Submission guidelines

• Make sure that you create one account single account for your team, and submit runs through that account only.
• The last file submitted to the leaderboard will be considered as the final submission.
• The file with your predictions should be called subtask1[A,B]_[lang].tsv, where A or B refer to the specific subtask and lang can be arabic, english or spanish. Get sure to set .tsv as the file extension; otherwise, you will get an error on the leaderboard. Subtasks are 1A and 1B. For subtask 1A, there are two languages (Arabic, and English). For subtask 1B, there are three languages (Arabic, Spanish, and English). For instance, a submission file for task 1B Arabic should be subtask1B_arabic.tsv.
• You have to zip the tsv, zip subtask1B_arabic.zip subtask1B_arabic.tsv and submit it through the codalab page.
• You have to include the team name and the description of your method each submission. Your team name must EXACTLY match the one used during the CLEF registration.
• You are allowed to submit max 200 submissions per day for each subtask.
• We will keep the leaderboard private until the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.

Submission Site

Task 1: Codalab

Contact

Please join the Slack channel for any queries: https://join.slack.com/t/newworkspace-qgd1635/shared_invite/zt-1k9lnm0ys-aMeEmUtGY0tfW5HbkXHPJg

Slack Channel (task1-check-worthiness): https://ct-23-participants.slack.com

Or send an email to: clef-factcheck@googlegroups.com

Leaderboard

All baselines are random systems.

1A Arabic

* Submissions without position were submitted after the deadline.

1A English

* Submissions without position were submitted after the deadline.

1B Arabic

* Submissions without position were submitted after the deadline.

1B English

* Submissions without position were submitted after the deadline.

1B Spanish

* Submissions without position were submitted after the deadline.

Organizers

• Firoj Alam, Qatar Computing Research Institute, HBKU
• Alberto Barrón-Cedeño, Università di Bologna, Italy
• Gullal S. Cheema, TIB – Leibniz Information Centre for Science and Technology
• Sherzod Hakimov, University of Potsdam
• Maram Hasanain, Qatar Computing Research Institute, HBKU
• Chengkai Li, The University of Texas at Arlington
• Rubén Miguez, Newtral, Spain
• Hamdy Mubarak, Qatar Computing Research Institute, HBKU
• Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence
• Gautam Kishore Shahi, University of Duisburg-Essen
• Wajdi Zaghouani, HBKU