Definition

In this task, the systems are challenged to retrieve a set of authority Twitter accounts for a given rumor propagating in Twitter.

Given a tweet stating a rumor, a model has to retrieve a ranked list of authority Twitter accounts that can help verify the rumor; i.e. they may tweet evidence that supports or denies the rumor.

This task is offered in Arabic.

Datasets

The training dataset is available here

It comprises 150 rumors expressed in tweets associated with 1,044 authorities’ Twitter accounts, a user collection of 395,231 Twitter accounts, and about 1.2M unique Twitter lists.

Evaluation

The official evaluation measure is P@5 to evaluate how systems are able to retrieve authorities at the top of a short retrieved list.

Other measures, such as P@1 and nDCG@5, will also be used.

Submission

Scorers, Format Checkers, and Baseline Scripts

All scripts can be found on gitlab at CheckThat! Lab Task 5 repository

Submission guidelines

To submit your runs (the output of your system), please use the form sent to all participants of Task 5 over email for each run. Your runs must be in the output format specified in the task repository. The submission form will be closed on May 10th.

Each team can submit up to 3 runs based in the following guidelines:

• For each run, you will have to explicitly indicate if it is primary or secondary.
• One and only one run must be primary and it will be used for main comparison to other teams.
• For each run, you will have to explicitly indicate if external data was used for training.
• Maximum of one run can use external data.

Leaderboard

The official metric is P@5

* Submissions with a + sign include task organisers

Organizers

• Tamer Elsayed, Qatar University, Qatar
• Fatima Haouari, Qatar University, Qatar
• Zien Sheikh Ali, Qatar University, Qatar