Evaluating Text Preprocessing Effects on Indonesian Sentiment Classification of Islamic Boarding School Messages

Authors

  • Arfan Hidayatullah Universitas Islam Negeri Maulana Malik Ibrahim, Indonesia
  • Zainal Abidin Universitas Islam Negeri Maulana Malik Ibrahim, Indonesia

DOI:

https://doi.org/10.54518/rh.6.5.2026.2023

Keywords:

Ablation Study, Indonesian Language, Naive Bayes, Sentiment Analysis, Support Vector Machine, Text Preprocessing

Abstract

Islamic boarding schools manage large volumes of digital messages from students’ guardians and communities, making it difficult to identify stakeholder concerns. This study evaluated six text-preprocessing steps in Indonesian three-class sentiment classification and compared Multinomial Naive Bayes (NB) with Linear Support Vector Machine (SVM). The experiment used 2,257 messages from a pesantren communication channel. After identifiers and incomplete records were removed, three annotators labeled messages as positive, neutral, or negative; a fourth resolved cases without a majority. Agreement was measured using Fleiss’ kappa. Steps were examined through leave-one-step-out ablation using unigram TF-IDF, five-fold stratified cross-validation, held-out testing, learning curves, and Wilcoxon tests. Cross-validation accuracy ranged from 0.8155 to 0.8238 for NB and 0.8100 to 0.8133 for SVM. On full-pipeline test data, accuracy was 0.8540 for NB and 0.8208 for SVM. Removing individual steps produced no substantial gains. No statistically significant difference between classifiers was detected. NB performed better overall but tended to overpredict neutral messages; both models struggled with this class. The findings support the full pipeline as a baseline and identify annotation consistency and neutral-class handling as priorities for further improvement.

Downloads

Download data is not yet available.

References

Adriani, M., Asian, J., Nazief, B., Tahaghoghi, S. M., & Williams, H. E. (2007). Stemming Indonesian: A confix-stripping approach. ACM Transactions on Asian Language Information Processing (TALIP), 6(4), 1-33. https://doi.org/10.1145/1316457.1316459.

Ahmadian, H., Abidin, T. F., Riza, H., & Muchtar, K. (2024). Hybrid models for emotion classification and sentiment analysis in Indonesian language. Applied Computational Intelligence and Soft Computing, 2024(1), 282-297. https://doi.org/10.1155/2024/2826773.

Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297. https://doi.org/10.1007/BF00994018.

Cunha, W., Mangaravite, V., Gomes, C., Canuto, S., Resende, E., Nascimento, C., ... & Gonçalves, M. A. (2021). On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study. Information Processing & Management, 58(3), 1024-1041. https://doi.org/10.1016/j.ipm.2020.102481.

Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(1), 1–30.

Febrilliani, J. S., & Wibowo, A. (2025). Twitter public sentiment towards vasectomy in Indonesia using SVM and Naïve Bayes. Research Horizon, 5(4), 1415–1424. https://doi.org/10.54518/rh.5.4.2025.756.

Geni, L., Yulianti, E., & Sensuse, D. (2023). Sentiment analysis of tweets before the 2024 elections in Indonesia using BERT language models. Jurnal Ilmiah Teknik Elektro Komputer dan Informatika, 9(3), 746–757. https://doi.org/10.26555/jiteki.v9i3.26490.

Haddi, E., Liu, X., & Shi, Y. (2013). The role of text pre-processing in sentiment analysis. Procedia Computer Science, 17(1), 26–32. https://doi.org/10.1016/j.procs.2013.05.005.

Hidayat, R. A., Khasanah, I. N., Putri, W. C., & Mahendra, R. (2021). Feature-rich classifiers for recognizing textual entailment in Indonesian. Procedia Computer Science, 189(1), 148-155. https://doi.org/10.1016/j.procs.2021.05.094.

Jiwandono, T. A., Achmad, M. S. H., & Suhirman. (2026). Fine-tuned IndoBERT for stock market sentiment analysis: Evidence from CNBC Indonesia news. Indonesian Journal of Electrical Engineering and Computer Science, 42(3), 774–785. http://doi.org/10.11591/ijeecs.v42.i3.pp774-785.

Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In Proceedings of the 10th European Conference on Machine Learning (Lecture Notes in Computer Science, Vol. 1398, pp. 137–142). Berlin: Springer.

Kamal, M. R., Noviyanto, K., Subhan, H., & Afiana, L. (2024). Sentiment analysis on social media and stakeholders about negative issues among Islamic boarding school community in Indonesia. Jurnal Theologia, 35(2), 261–280. http://dx.doi.org/10.21580/teo.2024.35.2.23164.

Kobayashi, V. B., Mol, S. T., Berkers, H. A., Kismihók, G., & Den Hartog, D. N. (2018). Text classification for organizational researchers: A tutorial. Organizational Research Methods, 21(3), 766-799. https://doi.org/10.1177/1094428117719322.

Koto, F., Rahimi, A., Lau, J. H., & Baldwin, T. (2020). IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 757–770). Barcelona: International Committee on Computational Linguistics. https://doi.org/10.18653/v1/2020.coling-main.66.

Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), 150-167. https://doi.org/10.3390/info10040150.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310.

Latief, A. D., Jarin, A., Uliniansyah, M. T., Nurfadhilah, E., & Afra, D. I. N. (2023). A proven sentiment annotation guideline for Indonesian Twitter data. In 2023 International Conference on Computer, Control, Informatics and Its Applications (IC3INA) (pp. 31–36). New York: IEEE. https://doi.org/10.1109/IC3INA60834.2023.10285807.

Liu, B. (2012). Sentiment analysis and opinion mining. California: Morgan & Claypool Publishers.

Martinčić-Ipšić, S., Miličić, T., & Todorovski, L. (2019). The influence of feature representation of text on the performance of document classification. Applied Sciences, 9(4), 743-755. https://doi.org/10.3390/app9040743.

McCallum, A., & Nigam, K. (1998). A comparison of event models for Naive Bayes text classification. In AAAI-98 Workshop on Learning for Text Categorization (pp. 41–48). New York: AAAI Press.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(23), 2825–2830.

Putri, G. K., Sujaini, H., & Ulumi, D. I. (2025). Perbandingan algoritma Support Vector Machine (SVM) dan Naive Bayes pada analisis sentimen Bahasa Jawa dan Sunda. JUSTIN (Jurnal Sistem dan Teknologi Informasi), 13(2), 200-211. https://doi.org/10.26418/justin.v13i2.88285.

Robertson, S. (2004). Understanding inverse document frequency: on theoretical arguments for IDF. Journal of Documentation, 60(5), 503-520. https://doi.org/10.1108/00220410410560582.

Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513–523. https://doi.org/10.1016/0306-4573(88)90021-0.

Supian, A., Tri Revaldo, B., Marhadi, N., Efrizoni, L., & Rahmaddeni, R. (2024). Perbandingan kinerja Naïve Bayes dan SVM pada analisis sentimen Twitter Ibu Kota Nusantara. Jurnal Ilmiah Informatika, 12(1), 15–21. https://doi.org/10.33884/jif.v12i01.8721.

Tala, F. Z. (2003). A study of stemming effects on information retrieval in Bahasa Indonesia. Amsterdam: University of Amsterdam (Master’s thesis).

Uysal, A. K., & Gunal, S. (2014). The impact of preprocessing on text classification. Information Processing & Management, 50(1), 104–112. https://doi.org/10.1016/j.ipm.2013.08.006.

Vijayarani, S., Ilamathi, M. J., & Nithya, M. (2015). Preprocessing techniques for text mining-an overview. International Journal of Computer Science & Communication Networks, 5(1), 7-16.

Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6), 80–83. https://doi.org/10.2307/3001968.

Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., & Purwarianti, A. (2020). IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing (pp. 843–857). Suzhou: Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.aacl-main.85.

Zhou, X., Gururajan, R., Li, Y., Venkataraman, R., Tao, X., Bargshady, G., ... & Kondalsamy-Chennakesavan, S. (2020). A survey on text classification and its applications. In Web intelligence (Vol. 18, No. 3, pp. 205-216). Thousand Oaks: SAGE Publications. https://doi.org/10.3233/WEB-200442.

Downloads

Published

2026-10-25

How to Cite

Hidayatullah, A., & Abidin, Z. (2026). Evaluating Text Preprocessing Effects on Indonesian Sentiment Classification of Islamic Boarding School Messages . Research Horizon, 6(5), 2409–2422. https://doi.org/10.54518/rh.6.5.2026.2023

Similar Articles

<< < 65 66 67 68 69 70 71 72 73 74 > >> 

You may also start an advanced similarity search for this article.