Definition

The aim of this task is to determine whether a claim in a tweet and/or transcriptions is worth fact-checking. Typical approaches to make that decision require to either resort to the judgments of professional fact-checkers or to human annotators to answer several auxiliary questions such as “does it contain a verifiable factual claim?”, and “is it harmful?”, before deciding on the final check-worthiness label https://aclanthology.org/2021.findings-emnlp.56.pdf.

This year, we are offering multi-genre data: the tweets and/or transcriptions should be judged based solely on the text. The task is available in Arabic, English, and Spanish.

Datasets

Data for all languages are available here

Each instance is composed of only text, which could come from a tweet, the transcription of a debate or the transcription of speech.

Evaluation

This is a binary classification task. The official evaluation metric is F_1 over the positive class.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at CheckThat! Lab Task 1 repository

Submission guidelines

• Make sure that you create one account for each team, and submit runs through one account only.
• The last file submitted to the leaderboard will be considered as the final submission.
• Name of the output file has to be task1_lang.tsv with .tsv extension (e.g., task1_arabic.tsv); otherwise, you will get an error on the leaderboard. Three languages are possible (Arabic, English, and Dutch).
• You have to zip the tsv, zip task1_arabic.zip task1_arabic.tsv and submit it through the codalab page.
• It is required to submit the team name and method description for each submission. Your team name here must EXACTLY match that used during CLEF registration.
• You are allowed to submit max 200 submissions per day.
• We will keep the leaderboard private till the end of the submission period, hence, results will not be available upon submission. All results will be available after the evaluation period.

Submission Site

Task 1: Codalab

Contact

Please join the Slack channel for any queries: https://join.slack.com/t/ct-participants/shared_invite/zt-1me6ywqxu-0Q~r7vCm6gfmrmN9xuOO7A

Or send an email to: clef-factcheck@googlegroups.com

Leaderboard

All baselines are random systems.

Task 1 Arabic

Task 1 Dutch

Task 1 English

Organizers

• Firoj Alam, Qatar Computing Research Institute, HBKU, Qatar
• Maram Hasanain, Qatar Computing Research Institute, HBKU, Qatar
• Alberto Barrón-Cedeño, Università di Bologna, Italy
• Reem Suwaileh, HBKU, Qatar
• Chengkai Li, The University of Texas at Arlington, USA
• Rubén Miguez, Newtral, Spain
• Wajdi Zaghouani, HBKU, Qatar
• Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence, UAE
• Sanne Weering, University of Groningen, Netherlands
• Tommaso Caselli, University of Groningen, Netherlands