Sentiment Analysis using Bidirectional Encoder Representations from Transformers
DOI:
https://doi.org/10.53560/PPASA(61-2)870Keywords:
Sentiment Analysis, Tweets Dataset, BiLSTM, BERT ModelAbstract
In the contemporary digital landscape, a significant volume of data is generated through social networks such as Twitter, Facebook, and Instagram. This study presents a method for extracting sentiments from Twitter, focusing on two sentiment-based datasets: the Twitter and emotional sentiments datasets. After extraction and preprocessing, we employed three deep learning models: Recurrent Neural Networks (RNNs), Bidirectional Long Short-Term Memory (BiLSTM), and a pre-trained Bidirectional Encoder Representations from Transformers (BERT) model. We introduced Se-BERT, a model designed for emotional sentiment analysis. Our experiments showed that Se-BERT achieved accuracy levels of 97.29% for tweet sentiments (positive and negative) and 86.77% for emotional sentiments (joy, sadness, love, fear, anger, surprise). These results demonstrate that Se-BERT outperforms RNN and BiLSTM in terms of accuracy for sentiment analysis, thereby significantly enhancing information retrieval and providing a deeper understanding of user behaviour.
References
E. Adamopoulou, and L. Moussiades. Chatbots: History, technology, and applications. Machine Learning with Applications 2: 100006 (2020).
K. Machova, M. Szaboova, J. Paralic, and J. Micko. Detection of emotion by text analysis using machine learning. Frontiers in Psychology 14: 1190326 (2023).
E. Cambria, and B. White. Jumping NLP curves: A Review of Natural Language Processing Research. IEEE Computational Intelligence Magazine 9(2): 48-57 (2014).
J.C. Pereira-Kohatsu, L. Quijano-Sanchez, F. Liberatore, and M. Camacho-Collados. Detecting and Monitoring Hate Speech in Twitter. Sensors 19(21): 4654 (2019).
F.M. Shiri, T. Perumal, N. Mustapha, and R. Mohamed. A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU. Preprint ArXiv 1: 2305.17473 (2023).
K. Du, F. Xing, R. Mao, and E. Cambria. Financial Sentiment Analysis: Techniques and Applications. ACM Computing Surveys 56(9): 1-42 (2024).
U. Singh, K. Abhishek, and H.K. Azad. A Survey of Cutting-edge Multimodal Sentiment Analysis. ACM Computing Surveys 56(9): 1-36 (2024).
A. Alslaity, and R. Orji. Machine learning techniques for emotion detection and sentiment analysis: current state, challenges, and future directions. Behaviour and Information Technology 43(1): 139-164 (2024).
C.R. Sugimoto, S. Work, V. Lariviere, and S. Haustein. Scholarly use of social media and altmetrics: A review of the literature. Journal of the Association for Information Science and Technology 68(9): 2037-2062 (2017).
J.P. Bharadiya. A Comprehensive Survey of Deep Learning Techniques Natural Language Processing. European Journal of Technology 7(1): 58-66 (2023).
Z. Jin, Y. Yang, and Y. Liu. Stock closing price prediction based on sentiment analysis and LSTM. Neural Computing and Applications 32: 9713-9729 (2020).
J. Zhao, Z. Dalin, Y. Xiao, L. Che, and M. Wang. User personality prediction based on topic preference and sentiment analysis using LSTM model. Pattern Recognition Letters 138: 397-402 (2020).
M. Li, W. Li, F. Wang, X. Jia, and G. Rui. Applying BERT to analyse investor sentiment in stock market. Neural Computing and Applications 33: 4663-4676 (2021).
M. Munikar, S. Shakya, and A. Shrestha. Fine-grained Sentiment Classification Using BERT. Preprint ArXiv 1: 1910.03474 (2019).
M. Singh, A.K. Jakhar, and S. Pandey. Sentiment analysis on the impact of coronavirus in social life using the BERT model. Social Network Analysis and Mining 11: 33 (2021).
M. Tripathi. Sentiment Analysis of Nepali COVID19 Tweets Using NB SVM and LSTM. Journal of Artificial Intelligence 3: 151-168 (2021).
T. Chen, R. Xu, Y. He, and X. Wang. Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Systems with Applications 72: 221-230 (2017).
M. Rhanoui, M. Mikram, S. Yousfi, and S. Barzali. A CNN-BiLSTM model for Document-Level Sentiment Analysis. Machine Learning and Knowledge Extraction 1(3): 832-847 (2019).
M.E. Basiri, S. Nemati, M. Abdar, E. Cambria, and U.R. Acharya. ABCDM: An Attention-based Bidirectional CNN-RNN Deep Model for sentiment analysis. Future Generation Computer Systems 115: 279-294 (2021).
X. Li, L. Bing, W. Zhang, and W. Lam. Exploiting BERT for End-to-End Aspect-based Sentiment Analysis. Preprint ArXiv 2: 1910.00883 (2019).
M. Hoang, O.A. Bihorac, and J. Rouces. Aspect-based sentiment analysis using bert. Proceedings of the 22nd Nordic Conference on Computational Linguistics, Turku, Finland, (September 30 – October 02, 2019) pp. 187-196 (2019).
M. Li, L. Chen, J. Zhao, and Q. Li. Sentiment analysis of Chinese stock reviews based on BERT model. Applied Intelligence 51: 5016–5024 (2021).
M. Heidari, and J.H. Jones. Using BERT to Extract Topic-Independent Sentiment Features for Social Media Bot Detection. 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, USA pp. 0542-0547 (2020).
E.F. Can, A. Ezen-Can, and F. Can. Multilingual Sentiment Analysis: An RNN-Based Framework for Limited Data. Preprint ArXiv: 1806.04511 (2018).
M.M. KazAnova. Sentiment140 dataset with 1.6 million tweets. Kaggle: (2017). https://www.kaggle.com/datasets/kazanova/sentiment140 (accessed 22 July 2023)
P. Praveen. Emotions dataset for NLP. Kaggle: (2020). https://www.kaggle.com/datasets/praveengovi/emotions-dataset-for-nlp (accessed 22 July 2023)
S. Cornegruta, R. Bakewell, S. Withey, and G. Montana. Modelling Radiological Language with Bidirectional Long Short-Term Memory Networks. Preprint ArXiv:1609.08409 (2016).
G. Xu, Y. Meng, X. Qiu, Z. Yu, and X. Wu. Sentiment Analysis of Comment Texts based on BiLSTM. IEEE Access 7: 51522–51532 (2019).
K. Das, M. Shehryar, F. Abid, M. Ashraf, M. Adil, S. Inam, E.U. Haq, and H. Mushtaq. Impact of using e-learning tools on Student's Psychological Health during covid-19. VFAST Transactions on Software Engineering 9(3): 120-127 (2021).
W. Yue, and L. Li. Sentiment Analysis using Word2vec-CNN-BiLSTM Classification. Seventh International Conference on Social Networks Analysis, Management and Security (SNAMS), Paris, France (December 14–16, 2020) pp. 1-5 (2020).
L. Zhao, L. Li, X. Zheng, and J. Zhang. A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts. Preprint ArXiv: 2001.05326 (2020).
Y. Song, J. Wang, Z. Liang, Z. Liu, and T. Jiang. Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference. Preprint ArXiv: 2002.04815 (2020).
A. Karimi, L. Rossi, and A. Prati. Adversarial Training for Aspect-Based Sentiment Analysis with BERT. Preprint ArXiv: 2001-11316 (2021).
S.M. Rezaeinia, R. Rahmani, A. Ghodsi, and H. Veisi. Sentiment analysis based on improved pre-trained word embeddings. Expert Systems with Applications 117: 139–147 (2019).
