What if Ground Truth is Subjective? Personalized Deep Neural Hate Speech Detection

Abstract

A unified gold standard commonly exploited in natural language processing (NLP) tasks requires high inter-annotator agreement. However, there are many subjective problems that should respect users’ individual points of view. Therefore, in this paper, we evaluate three different personalized methods for the task of hate speech detection. Our user-centered techniques are compared to the generalizing baseline approach. We conduct our experiments on three datasets including single-task and multi-task hate speech detection. For validation purposes, we introduce a new data split strategy, which prevents data leakage between training and testing. To better understand the behavior of the model for individual users, we carried out personalized ablation studies. Our experiments revealed that all models leveraging user preferences in any case provide significantly better results than most frequently used generalized approaches. This supports our general observation that personalized models should always be considered in all subjective NLP tasks, including hate speech detection.