Towards an Automated Essay Evaluation System NLP Based Text Embeddings and Similarity Metrics

Abstract

This study aims to develop an automatic essay answer assessment system based on Natural Language Processing (NLP) to reduce the time and effort required for evaluation. The system uses Cosine Similarity and Manhattan Distance as evaluation metrics and implements two text embedding methods—Term Frequency-Inverse Document Frequency (TF-IDF) and Bag of Words (BoW)—to represent the user’s answer text. The methodology begins with text processing and pre-processing, followed by embedding and similarity calculation between the user’s answer and the reference text to generate an evaluation score categorized into three levels: good, sufficient, and poor. Based on Cohen’s Kappa analysis, the kappa value for Cosine Similarity reaches 0.78, indicating high agreement between the Cosine TF-IDF and Cosine BoW methods. In contrast, Manhattan Distance yields a kappa value of -0.05, indicating a discrepancy between the two Manhattan-based methods. The evaluation results suggest that Cosine Similarity is more suitable, whereas Manhattan Distance is not relevant for this task. At the modeling stage, the best classification models are Decision Tree and Random Forest, each achieving an accuracy of 96.67%. Although Random Forest demonstrates a higher AUC than Decision Tree, it requires a longer training time. Overall, the system is considered effective for assessing essay answers with both purpose and consistency, offering potential applications in the field of education