Higher Education Category Submission

388 – An Intelligent Auxiliary System for Test Paper Analysis and Assessment Based on AI and Machine Learning

In education assessment, examination paper analysis, the examination of examination questions, and the control of final examination scores are important links in improving teaching quality. The traditional method of manually analyzing the difficulty of examination papers through statistics to design examination papers has relatively large errors. It may not be able to meet the reasonable distribution of the difficulty of examination questions and the control of the average score, and it will consume a large amount of manpower and time.

This project utilizes natural language processing and machine learning technologies to extract Knowledge points of questions, quantify the Difficulty distribution and Historical score data, and generate detailed analysis models of multiple examination papers for educators, so as to assist in the optimization of teaching and assessment.
This project analyzes students' test scores and examination questions during a semester of study, which is used for the optimization of the difficulty of the final examination paper and the control of the class average score.

Knowledge points of questions: The type of knowledge points of questions
Difficulty distribution: The proportion of scores of questions of different difficulty levels in the examination paper
Historical score data: The average score of students for each examination paper

A summer internship as a teaching assistant for one of the school vacations at my high school motivated this project. Reading dictations, tutoring, marking IELTS listening comprehension, and testing students' level were part of the job. Manual analysis was time-consuming and prone to errors, however, as it was not feasible to estimate students' actual learning outcomes to any degree of accuracy or plan lessons optimally in terms of time. First-hand experience thus witnessed the inefficiency of the traditional manual system of analysis, and this motivated the development of an AI-based system to automate and optimize the design of exams and analysis of them. The project assumes that natural language processing (NLP) and machine learning (ML) can be leveraged to identify knowledge points to select, compute difficulty distributions, and estimate scores so that the difficulty of the exam can be controlled by the teacher and teaching methods optimized with student outcomes.
By using historic scoring records and test metadata, the system can minimize human error, save time, and ensure the fairness of the test. The success is only a matter of time because AI technologies have the ability to process massive amounts of data to analyze patterns and trends tens or hundreds of times more potent than human intelligence.

N/A

There are three fundamental modules in system architecture: Multi-Format Exam Parsing, Quantitative Exam Statistical Analysis, and Target-Oriented Intelligent Question Generation. Developed with Tkinter as a GUI tool, the platform imports PDF, Word, and TXT, imports text using PyPDF2 and python-docx, and analyzes the data using Pandas. NLP (e.g., Moonshot-v1-8k) is used to predict the difficulty of knowledge points, and scikit-learn linear regression models are used to predict difficulty-to-score correlation.
To generate new target-based features for question generation, the system predicts weighted difficulty coefficients (DCW) from target average scores using regression models.
Moonshot AI creates new tests based on syllabus requirements, adjusts difficulty and knowledge distribution. Project Timeline comprises Phase 1 (model training and training data preparation, Q1 2025), Phase 2 (API integration and GUI testing, Q2 2025), and Phase 3 (usability feedback on scifi writing and scalability improvement, Q3 2025). Performance metrics are prediction accuracy (Mean Squared Error, R²), user satisfaction survey, and time benefit over existing practices.

We use AI to assist in the analysis of examination papers. Through Natural Language Processing (NLP) technology, we extract knowledge points and quantify the difficulty, solving the problems of inefficiency and errors in traditional manual analysis. Secondly, we employ quantitative statistical analysis to calculate the weighted difficulty coefficient of each question, and conduct a linear regression analysis to explore the relationship between difficulty and scores. Then, we present the results in the form of charts. Through visual analysis, it helps teachers identify teaching problems, adjust teaching strategies, and improve the quality of teaching.
The project is innovative in that it end-to-end automates exam life cycle management from analysis to generation. Compared to existing tools, it combines NLP-based knowledge extraction and ML-based predictive modeling to dynamically manage difficulty levels of exams. For instance, it identifies knowledge gaps by comparing predicted and received scores and auto-generates customized exams to bridge gaps. The innovation is the adaptive question generator module that produces two kinds of exams (e.g., "BAD" and "GOOD") with varying difficulty levels. "BAD" is mastery in simple formulaic application, while "GOOD" is mastery in multi-concept analysis and experiment design and therefore suits different students. This personalization enhances effectiveness in education and fairness in that it suits novice and expert students.

For scalability, we can further design user-friendly APP or web interfaces to facilitate users in inputting and analyzing examination paper data. At the data level, by integrating multi-dimensional data such as the correct rate, we can further optimize the predictive ability of the model. To scale it, the system will be migrated from a desktop application to a cloud-based web/app interface with simultaneous multi-user access. Community contribution will be facilitated with a crowdsourced question bank so that it can be more adaptable in subject and exam board. Technical limitations, such as limitations in image generation, will be eliminated by adding Midjourney API for basic graphics (Q4 2025) and speech-to-text for language exams (Q2 2026).
For sustainability, we can improve the stability and prediction accuracy of the model by integrating algorithms such as Random Forest and XGBoost. Using open-source technologies can reduce development and operation costs and minimize resource consumption. We can also collect data and information through user feedback and cooperation with educational institutions, and continuously improve the functions of the system. This not only reduces costs but also ensures its long-term availability. Environmental sustainability is achieved through the use of less paper and green cloud storage. There are mechanisms of feedback (e.g., Likert-scale surveys, open comments) through which, in long-term adoption, the system would be upgraded to the next version. Seamless integration with edtech infrastructure will bring the tool into existing work flows with the ability to be adaptive to varying parameters of learning.

The program we developed can promote educational equity and help regions with backward educational resources access question resources more conveniently. At the same time, it can also assist novice teachers in designing examination papers of reasonable difficulty, alleviating the problem of shortage of examination resources. This program liberates human resources. By using AI to process a large amount of data and information, it can not only avoid human errors but also improve efficiency. Moreover, it provides references for policy formulation, curriculum revision, and textbook compilation, enhancing the distribution efficiency of educational resources. This program can also be extended for other purposes, such as supporting personalized teaching, generating targeted examination papers based on students' weaknesses, and meeting the learning needs of different students.
To measure the impact of this program, several social impact assessment indicators have been established. Firstly, the User Coverage indicator focuses on evaluating the usage of the system in different regions and schools, which helps to understand its penetration and reach. Secondly, the Teaching Improvement Effect assesses the actual effect of the system through students' grades and teachers' feedback, directly gauging its influence on educational outcomes. Thirdly, the Resource Conservation Indicator evaluates the performance of the system in reducing teachers' workload and resource consumption, highlighting its efficiency benefits.
The project is addressing education inequity by introducing data-driven exam construction technology to new teachers and resource-constrained schools, decreasing dependence on resource-hungry manual methods. It is addressing United Nations SDG 4 (Quality Education) through the enablement of inclusive assessment practice and learning at scale. Control accuracy of mean scores, teacher time saved (e.g., 50% time saving through automation), and student performance improvement (e.g., 20% decrease in predicted to actual score difference) are social impact metrics. AI decision-making transparency (e.g., visualisation of DCW-score relations) and ethical use of data are accountability metrics. User feedback sessions and audits on a regular basis will give responsiveness to the needs of the community, in particular the needs of marginalised communities with limited access to high quality assessments.

Personal Information Collection Statement (PICS):
1. The personal data collected in this form will be used for activity-organizing, record keeping and reporting only. The collected personal data will be purged within 6 years after the event.
2. Please note that it is obligatory to provide the personal data required.
3. Your personal data collected will be kept by the LTTC and will not be transferred to outside parties.
4. You have the right to request access to and correction of information held by us about you. If you wish to access or correct your personal data, please contact our staff at lttc@eduhk.hk.
5. The University’s Privacy Policy Statement can be access at https://www.eduhk.hk/en/privacy-policy.

• I have read and agree to the competition rules and privacy policy.