Robustness of fake news detectors

Details

Supervisors

Standaert, François-Xavier

Faculty

Ecole polytechnique de Louvain

Degree label

Master [120] : ingénieur civil en mathématiques appliquées

Abstract

The goal of our study is to evaluate the robustness of different fake news detectors, i.e., to determine how difficult it is to slightly modify fake articles so that the detector label them as reliable. For this, we first built a database of articles from both reliable and non-reliable media outlet. On that database, we constructed 5 fake news classifiers using well-known machine learning techniques. We studied the robustness of two of them, chosen for their high performance and for having inherently different classification strategies: logistic regression combined with tf-idf ignores the order of the words, while Long Short-Term Memory neural network (LSTM) with word embedding doesn’t. We evaluated robustness under 3 scenarios of attack: 1) low semantic and syntactic constraints with black box knowledge and full automation, 2) strong semantic and syntactic constraints with human intervention and white box knowledge, and 3) the latter with black box knowledge. We characterize robustness by the proportion of fake articles that we can successfully change so that they become labeled as reliable. The logistic regression classifier is robust: the proportion is less than 32 percent, under the 3 scenarios of attack. The LSTM classifier is robust to the low constraint attack without human intervention (22 percent), but less so when human intervention is allowed: 3 out of 5 articles were successfully transformed into adversarial articles. We believe however that this last result could be improved with further study. Finally, as part of our robustness study, we evaluated transferability, i.e., how the adversarial articles created for these two classifiers are classified by the other three classifiers we created at the begin of our study. The naive bayes classifier was the most robust: all of the adversarial articles were still classified as fake. We conclude that fake news detectors are useful to prevent the spread of fake news.