Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority of News Articles and Their Sources

Talks

Overview of CheckThat! 2023, presented at ECIR 2023 in Dublin, Ireland

Tasks

Task 1 Check-Worthiness in Multimodal and Unimodal Contents
Task 2 Subjectivity in News Articles
Task 3 Political Bias of News Articles and News Media
Task 4 Factuality of Reporting of News Media
Task 5 Authority Finding in Twitter

Important Dates

~~November 2022~~: Lab registration opens
~~December 2022~~: Release of the training materials
~~28 April 2023~~: Lab registration closes
3 May 2023: Beginning of the evaluation cycle (test sets release)
10 May 2023 (23:59 AOE): End of the evaluation cycle (run submission)
5 June 2023: Deadline for the submission of working notes
23 June 2023: Notification of acceptance of working notes
7 July 2023: Deadline for submission of camera-ready working notes
July 2023: Preview of working notes
18-21 September: CLEF 2023 Conference in Thessaloniki

Proceedings

Instructions to Prepare Your Paper

The title of paper should be in the following format: <Team Name> at CheckThat! 2023: <Descriptive title>

For example, team LamdaX would their title as follows: LamdaX at CheckThat! 2023: Cool Models for Checkworthiness Identification

Templates: The working notes are prepared using the CEUR-WS templates. Instructions and templates are both available at the CLEF 2023 website.

Overview paper: We are working on the preparation of the overview papers ourselves (one for the full lab and one per each task). Please stay tuned for the details. Meanwhile, you can consider to cite the ECIR 2023 paper about the lab.

Recent Updates

18th May, 2023: Leaderboards and test gold standards for all tasks have been published. Check on the Task section of this website and the corresponding git projects.
10th May, 2023: Submissions for all tasks are allowed **until 11th May (23:59 AOE)** due to problems during the submission by a number of teams. Notice that runs submitted after the original deadline (10th May, 23:59 AOE) **will be flagged accordingly** in the official leaderboards and in the overview papers. If you **do NOT want your submission to be flagged as late**, do submit by 10 May 2023 (23:59 AOE), as originally scheduled!
3rd May, 2023: The test sets for all tasks have been released. Predict!
28th April May, 2023: The registration to the shared task was closed.

Discussion

There are two channels for discussion and doubts:

Please join our discussion group https://groups.google.com/g/clef-factcheck to receive announcements and participate in discussions.

A slack channel is available as well.

Coordinators

General: Alberto Barrón-Cedeño, Università di Bologna, Italy
Task 1: Firoj Alam
Task 2: Andrea Galassi, Università di Bologna, Italy
Task 3: Giovanni Da San Martino, Università di Padova, Italy
Task 4: Preslav Nakov, MBZUAI, UAE
Task 5: Tamer Elsayed, Qatar University, Qatar

PC chairs

Tamer Elsayed, Qatar University, Qatar
Fatima Haouari, Qatar University, Qatar

Citation

There are various papers associated with this edition of the lab. So far, the ECIR 2023 is the only one published (at the bottom). Details for the papers specific to each task as well as an overall overview will be posted here as they come out.

Bib entries for each paper are included here. For your convenience, the bib file is available as well.

CLEF 2023 LNCS

Cite this paper to refer to the whole CheckThat! 2023 lab.

As of 23rd May, we are still defining this paper. Stay tuned!

CLEF 2023 Working notes

Task 1

Cite this paper to refer to Task 1 of the CheckThat! 2023 lab.

Alam, Barrón-Cedeño, Cheema, Hakimov, Hasanain, Li, Míguez, Mubarak, Shahi, Zaghouani, and Nakovet. Overview of the CLEF-2023 CheckThat! Lab Task 1 on Check-Worthiness in Multimodal and Multigenre Content. In Proceedings of the Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum. Thessaloniki, Greece

@InProceedings{clef-checkthat:2023:task1,
author = {
	and Alam, Firoj
	and Barr\'{o}n-Cede\~{n}o, Alberto
	and Cheema, Gullal S.
	and Hakimov, Sherzod
	and Hasanain, Maram
	and Li, Chengkai
	and M\'{\i}guez, Rub\'{e}n
	and Mubarak, Hamdy,
	and Shahi, Gautam Kishore
	and Zaghouani, Wajdi
	and Nakov, Preslav},
	title = "Overview of the {CLEF}-2023 {CheckThat}! Lab Task 1 on 
Check-Worthiness in Multimodal and Multigenre Content",
	year = {2023},
	booktitle = "Working Notes of CLEF 2023---Conference and Labs of the 
Evaluation Forum",
	series = {CLEF~'2023},
	address = {Thessaloniki, Greece},
}

Task 2

Cite this paper to refer to Task 2 of the CheckThat! 2023 lab. Overview of the CLEF-2023 CheckThat! Lab Task 2 on Subjectivity in News Articles. Galassi, Ruggeri, Barrónn-Cedeño, Alam, Caselli, Kutlu, Struss, Antici, Hasanain, Köhler, Korre, Leistra, Muti, Siegel, Turkmen, Wiegand, and Zaghouani. In Proceedings of the Working Notes of CLEF 2023 - Conference and Labs of the Evaluation Forum. Thessaloniki, Greece

As of 23rd May, we are still defining this paper. Stay tuned!

@InProceedings{clef-checkthat:2023:task2,
author = {Andrea Galassi 
	and Federico Ruggeri
	and Alberto Barr\'{o}n-Cede\~{n}o
	and Firoj Alam
	and Tommaso Caselli
	and Mucahid Kutlu
	and Julia Maria Struss
	and Francesco Antici
	and Maram Hasanain
	and Juliane Köhler
	and Katerina Korre
	and Folkert Leistra 
	and Arianna Muti
	and Melanie Siegel
	and Mehmet Deniz Turkmen
	and Michael Wiegand
	and Wajdi Zaghouani},
	title = "Overview of the {CLEF}-2023 {CheckThat}! Lab Task 2 on Subjectivity 
in News Articles",
	year = {2023},
	booktitle = "Working Notes of CLEF 2023--Conference and Labs of the 
Evaluation Forum",
	series = {CLEF~'2023},
	address = {Thessaloniki, Greece},
}

Task 3

Cite this paper to refer to Task 3 of the CheckThat! 2023 lab.

Da San Martino, Alam, Hasanain, Nandi, Azizov and Nakov. Overview of the CLEF-2023 CheckThat! Lab Task 3 on Political Bias of News Articles and News Media. In Proceedings of the Working Notes of CLEF 2023-Conference and Labs of the Evaluation Forum. Thessaloniki, Greece

@InProceedings{clef-checkthat:2023:task3,
author = {Da San Martino, Giovanni
	and Alam, Firoj
	and Hasanain, Maram
	and Nandi, Rabindra Nath
	and Azizov, Dilshod
	and Nakov, Preslav},
	title = "Overview of the {CLEF}-2023 {CheckThat}! Lab Task 3 on Political 
Bias of News Articles and News Media",
	year = {2023},
	booktitle = "Working Notes of CLEF 2023--Conference and Labs of the 
Evaluation Forum",
	series = {CLEF~'2023},
	address = {Thessaloniki, Greece},
}

Task 4

Cite this paper to refer to Task 4 of the CheckThat! 2023 lab.

Nakov, Alam, Da San Martino, Hasanain, Nandi, Azizov, and Panayotov. Overview of the CLEF-2023 CheckThat! Lab Task 4 on Factuality of Reporting of News Media. In Proceedings of the Working Notes of CLEF 2023-Conference and Labs of the Evaluation Forum. Thessaloniki, Greece

@InProceedings{clef-checkthat:2023:task4,
author = {Nakov, Preslav
	and Alam, Firoj
	and Da San Martino, Giovanni
	and Hasanain, Maram
	and Nandi, Rabindra Nath
	and Azizov, Dilshod
	and Panayotov, Panayot},
	title = "Overview of the {CLEF}-2023 {CheckThat}! Lab Task 4 on Factuality 
of Reporting of News Media",
	year = {2023},
	booktitle = "Working Notes of CLEF 2023--Conference and Labs of the 
Evaluation Forum",
	series = {CLEF~'2023},
	address = {Thessaloniki, Greece},
}

Task 5

Cite this paper to refer to Task 5 of the CheckThat! 2023 lab.

Haouari, Sheikh Ali and Elsayed. Overview of the CLEF-2023 CheckThat! Lab Task 5 on Authority Finding in Twitter. In Proceedings of the Working Notes of CLEF 2023-Conference and Labs of the Evaluation Forum. Thessaloniki, Greece

@InProceedings{clef-checkthat:2023:task5,
author = {Haouari, Fatima
	and Sheikh Ali, Zien 
	and Elsayed, Tamer
	},
	title = "Overview of the {CLEF}-2023 {CheckThat}! Lab Task 5 on Authority 
Finding in Twitter",
	year = {2023},
	booktitle = "Working Notes of CLEF 2023--Conference and Labs of the 
Evaluation Forum",
	series = {CLEF~'2023},
	address = {Thessaloniki, Greece},
}

ECIR 2023

So far this is the only paper published already. You can cite it when referring to the whole lab.

[1] Barrón-Cedeño, Alam, Caselli, Da San Martino, Elsayed, Galassi, Haouari, Ruggeri, Struss, Nandi, Cheema, Azizov, Nakov (2023). The CLEF-2023 CheckThat! Lab: Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority. In: Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13982. Springer, Cham. https://doi.org/10.1007/978-3-031-28241-6_59

@InProceedings{10.1007/978-3-031-28241-6_59,
	author="Barr{\'o}n-Cede{\~{n}}o, Alberto
		and Alam, Firoj
		and Caselli, Tommaso
		and Da San Martino, Giovanni
		and Elsayed, Tamer
		and Galassi, Andrea
		and Haouari, Fatima
		and Ruggeri, Federico
		and Struss, Julia Maria
		and Nandi, Rabindra Nath
		and Cheema, Gullal S.
		and Azizov, Dilshod
		and Nakov, Preslav",
	title="The CLEF-2023 CheckThat! Lab: Checkworthiness, Subjectivity, Political Bias, Factuality, and Authority",
	editor="Kamps, Jaap
		and Goeuriot, Lorraine
		and Crestani, Fabio
		and Maistro, Maria
		and Joho, Hideo
		and Davis, Brian
		and Gurrin, Cathal
		and Kruschwitz, Udo
		and Caputo, Annalina",
	booktitle="Advances in Information Retrieval",
	year="2023",
	publisher="Springer Nature Switzerland",
	address="Cham",
	pages="506--517",
	isbn="978-3-031-28241-6"
}

CLEF 2023

This edition of the CheckThat! lab is held within CLEF 2023, part of the CLEF initiative.

Lab Registration

The lab registration is done through the CLEF 2023 website.

Contents