A DISTILBERT-Driven Course-Aware Retrieval-Augmented Framework for Educational Question Answering

Abstract

With the advent of artificial intelligence and natural language processing, new possibilities arose to build intelligent educational question-answering systems to provide relevant academic information to learners through natural language interaction. However, traditional approaches to question-answering could be ineffective in identifying what information a learner is interested in and therefore finding the relevant knowledge chunks in the course. In order to overcome the above issue, this paper introduces a Course-Aware Retrieval-Augmented Generation Framework Employing DistilBERT-based Intent Classification for Intelligent Questions Answering in Education. The proposed framework consists of two modules: a light-weight transformer-based intent classification module and a retrieval module based on course-awareness. This provides a structured mapping between the learner's question, question information category, and course information. It employs the Educational Questions & Answers dataset (12,571 question-answer pairs) that has been cleaned to get 11,702 valid records. The questions were broken down into five educational intents: level, price, generic, free/paid, enrollment, subject, and beginner suitability, excluding the generic category, training the classifiers. The 18,720 labeled data samples in the stratified dataset were split into both a training and validation subset and DistilBERT was fine tuned for intent classification. The obtained accuracy, precision, recall, F1-score at the selected first training epoch show that the classification of the defined educational intent categories has been accomplished in a very effective way with precision of 99.9466%, recall of 99.9563% and F1 score of 99.9541%. Based on the findings, we conclude that DistilBERT is a suitable layer for understanding the intention for educational retrieval-augmented question answering. However, this classification performance cannot be understood as end-to-end RAG answer accuracy as this is only based on intent-classification performance. The areas of future development will involve human annotated intent datasets, expanded educational domains, multilingual and code-mixed queries, improvements to hybrid retrieval, reranking, and complete end-to-end assessment of retrieval relevance, answer faithfulness, and answer quality.