DETERMINING THE AUTHENTICITY OF NEWS REPORTS USING HUMAN-IN-THE-LOOP TO COMBAT FAKE NEWS

Abstract

Annotated corpora play a vital role in training Natural Language Processing models. However, complex semantic annotation processes are costly, labor-intensive, and time-consuming, limiting the availability of resources for training Machine Learning and Deep Learning algorithms. To address this challenge, our work introduces a methodology based on the human-in-the-loop approach for semi-automating complex tasks. Applied this methodology to create a news reliability dataset to combat disinformation and fake news effectively. By employing this methodology, obtained a high-quality resource that enhances annotator efficiency and speed, requiring fewer examples. The methodology comprises three incremental phases, resulting in the creation of the news dataset. We assess the quality of the annotations through time reduction (nearly 64% reduction in annotation time compared to fully manual annotation), evaluation of annotation consistency, inter-annotator agreement, and model performance when trained on the dataset. Our results demonstrate the suitability and effectiveness of our proposed model, achieving an Accuracy of 97% and an F-Score of 99%.