Definition

Given a set of news articles and a list of 23 persuasion techniques, including logical fallacies (straw man, red herring, bandwagon, …) and emotional manipulation techniques (loaded language, appeal to fear, name calling, …) that might be used to support flawed argumentation, the task consists of identifying the spans of texts in which each technique occurs. Text spans assigned with labels might also overlap. Therefore, this is a multi-label multi-class sequence tagging task.

For example, in the following excerpt from a news article

Manchin says Democrats acted like [34]babies[40] at the SOTU Personal Liberty Poll Exercise your right to vote.

Democrat West Virginia Sen. Joe Manchin says his colleagues’ refusal to stand or applaud during President Donald Trump’s State of the Union speech was disrespectful and a signal that [288]the party is more concerned with obstruction than it is with progress[357].

you are supposed to figure out that between characters 34th and 40th there is an instance of the Name Calling technique and between the 288th and 357th characters an instance of False Dilemma.

We share the annotation guidelines to give more detailed definitions, with examples, of the output classes for the task (check only the part related to persuasion techniques on that document).

Datasets

The datasets are available on this webpage. For technical reasons, you’ll need to make another registration on that page, but it is going to be quick. We thank you for the extra time you’ll have to spend on this.

Beware that this registration does not exempt you from registering to the main CLEF website (which is necessary if you want to write a paper describing your system that will appear in the CLEF conference).

Evaluation

Here is a brief description of the evaluation measures the scorers compute.

The task is a multi-label multi-class sequence tagging task. We modify the standard micro-averaged F1 to account for partial matching between the spans. In addition, an F1 value is computed for each persuasion technique.

In a nutshell, strong partial overlap will be given a full credit (lean approach), while in cases in which the overlap is less strong the partial credit is proportional to the intersection of the two spans, and it is normalized by the length of the ground truth.

The official score that will appear on Leaderboard will be computed using the 23 fine-grained persuasion technique labels.

Submission

The task has two phases:

• In the first phase we provide a training set with gold labels, a development set without gold labels, and a live leaderboard to see your progress as well as the ones of the other participants on the development set.
• In the second phase we’ll provide a test set without gold labels and the performance on the test set, which will be revealed only at the end of the competition, will be the official one of your team.

The data comprises various languages, the following table specifies which language will be available as training, development or test set:

Notice that the official evaluation will be on the data available on the test set, so for Arabic, Slovenian and Bulgarian will be a zero-shot-type task (whilst for English we provide data for training, development and test sets). Spanish, Greek and Georgian can be used to test the zero-shot capabilities of your systems on the development set.

Submissions may be made for any number of languages (even just one), we provide a separate leaderboard for each language.

Submission Site

https://propaganda.math.unipd.it/checkthat24_persuasion/ (notice that you need to register on this website as well in order to submit your predictions).

Organizers

The task is the result of the efforts of:

• Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence, UAE
• Jakub Piskorski, Polish Academy of Sciences, Poland
• Nicolas Stefanovitch, European Commission Joint Research Centre, Italy
• Giovanni Da San Martino, University of Padova, Italy
• Elisa Sartori, University of Padova, Italy
• Ricardo Campos, University of Beira Interior - Covilhã and INESC TEC - Porto, Portugal
• Senja Pollak, Jozef Stefan Institute, Slovenia
• Dimitar Dimitrov, Sofia University, Bulgaria
• Firoj Alam, Qatar Computing Research Institute, HBKU, Qatar
• Alípio Jorge, University of Porto - Porto and INESC TEC - Porto, Portugal
• Purificação Silvano, University of Porto - Porto and CLUP - Porto, Portugal
• Nuno Guimarães, University of Porto - Porto and INESC TEC - Porto, Portugal
• Ana Filipa Pacheco, University of Porto, Portugal
• Nana Yu, University of Porto, Portugal
• Ana Zwitter Vitez, University of Ljubljana, Slovenia
• Zoran Fijavž, Peace institute, Slovenia
• Nikolay Ribin, Sofia University, Bulgaria
• Ivan Koychev, Sofia University, Bulgaria
• Maram Hasanain, Qatar Computing Research Institute, HBKU, Qatar
• Fatema Ahmed, Qatar Computing Research Institute, HBKU, Qatar

Contact

Questions that are task 3 specific should be directed to checkthat24task3organisers@googlegroups.com.

For more general questions on the Checkthat! Lab (such as deadlines, paper format) send an email to: clef-factcheck@googlegroups.com