Open Category Submission

236 – AVEA: Automatic Verbal Exam Assessor for Medical Education Using GenAI

Overview:
The AVEA project seeks to transform medical education by introducing an innovative tool for automating the assessment of verbal examinations. This system will streamline the process of evaluating students' verbal responses during exams, providing a more consistent, objective, and efficient method for assessment. By integrating advanced technologies, AVEA will deliver real-time evaluations, allowing for more accurate feedback and improving the overall learning experience for medical students.
Objectives:
• To automate the evaluation of verbal exam responses, ensuring consistency and fairness in assessment.
• To improve the accuracy of evaluations by removing potential examiner biases and reducing human error.
• To provide immediate, actionable feedback to students, fostering enhanced learning and helping them better understand their strengths and areas for improvement.
• To reduce the administrative burden and resource requirements associated with conducting traditional verbal exams, making the process more efficient.
• To contribute to the overall improvement of medical education by offering a scalable solution that can be easily implemented across various institutions.
Technical Approach:
The AVEA system is designed to automate the process of assessing verbal exam responses by combining speech-to-text technology with advanced analytical techniques such as Generative Pre-trained Transformer (GPT), Speech Recognition (SR) and Natural Language Processing (NLP). Students will respond to exam questions verbally, and their answers will be transcribed in real-time. Once transcribed, the system will evaluate the content of the responses based on criteria such as accuracy, depth, clarity, and relevance. This approach allows for an immediate and objective assessment, reducing the potential for human error and subjectivity. The system is built with flexibility in mind, allowing for integration with existing exam platforms and providing educators with comprehensive reports to track student performance over time.
The provided Figure1 describes the workflow of the proposed system (Automatic Verbal Exam Assessor - AVEA) using AI technologies for automating verbal examinations.

Figure 1: AVEA system overview
Here's a detailed explanation of the AVEA system:
Workflow Description:
1. Exam Question Display:
o Initially, exam questions are presented to the student on a digital interface, such as a computer or tablet.
2. Student Verbal Response:
o The student reads the displayed question and answers verbally.
o The system captures this verbal answer using a microphone and converts the spoken audio into written text via Speech Recognition (SR) technology.
3. AI-Generated Answer (GPT):
o Simultaneously, the system feeds the displayed exam question into a Generative Pre-trained Transformer (GPT) model.
o The GPT model uses related educational resources and course materials to generate an ideal or reference textual answer to the same question.
4. Natural Language Processing (NLP) Analysis:
o Both the student's transcribed answer and the GPT-generated reference answer are sent to an NLP processing module.
o The NLP module evaluates and compares these two answers by analyzing their semantic content, depth, accuracy, and coherence.
5. Similarity Scoring:
o Based on the NLP analysis, the system calculates a similarity score between the student's response and the AI-generated ideal answer.
o A high similarity score (indicated by a green checkmark) suggests the student has answered accurately and comprehensively.
o A low similarity score (indicated by a red cross) implies inaccuracies or gaps in the student's response, prompting feedback for improvement.
Key Components:
• Student: the user participating in the verbal examination.
• Microphone (Speech Recognition): captures and converts spoken responses into text.
• Generative AI (GPT): produces model answers based on pre-existing course material.
• Natural Language Processing (NLP): evaluates textual similarity and relevance.
This automated assessment approach provides objective, unbiased, real-time scoring and immediate feedback, significantly enhancing the efficiency, reliability, and consistency of verbal examinations.
Benefits and Expected Outcomes:
• The automated system ensures that all responses are evaluated according to the same criteria, eliminating inconsistencies arising from different examiners or subjective interpretations.
• By relying on an objective, systematic evaluation process, the system increases the reliability of assessments, ensuring that students are assessed on the quality of their knowledge and reasoning rather than how well they can communicate it in a specific format.
• Students will receive real-time feedback on their responses, helping them identify strengths and weaknesses as they go through the exam. This allows them to learn from their mistakes instantly, improving their understanding and performance.
• The system reduces the need for manual grading, saving valuable time for instructors and allowing exams to be administered more quickly and at a larger scale. This makes accommodating more students easier and reduces the logistical burden of traditional exams.
• By automating the evaluation process, institutions can reallocate resources toward other critical educational activities, improving overall academic quality and efficiency.
• The system helps ensure that students are better prepared for real-world medical challenges, as they will have received consistent, high-quality feedback during their training, ultimately contributing to the production of more skilled healthcare professionals.
Potential Impact:
The AVEA project has the potential to significantly enhance medical education by standardizing and streamlining the verbal assessment process. It improves evaluations' consistency, accuracy, and efficiency while providing immediate, actionable feedback for students. By reducing administrative burdens and resource requirements, AVEA optimises educational resources and contributes to developing more competent healthcare professionals, ultimately improving patient care outcomes.

Medical education relies heavily on verbal assessments to evaluate students' practical knowledge, communication skills, and clinical reasoning. However, traditional verbal exams often face significant challenges, including examiner variability, subjectivity, and the potential for grading errors. These inconsistencies can result in unfair assessments that do not accurately reflect a student's abilities. Additionally, the process of scheduling and conducting these exams is time-consuming and resource-intensive, often requiring extensive administrative support and coordination. This creates logistical difficulties and increases operational costs for educational institutions.
These limitations can hinder accurately measuring students' clinical and professional competencies, impacting future patient care quality and the healthcare system. Without a reliable, consistent way to evaluate verbal responses, it is challenging to ensure that medical students are fully prepared for real-world clinical situations.
The AVEA project addresses these issues by automating the verbal assessment process. By leveraging advanced technology to provide real-time, objective evaluations, AVEA eliminates human biases and errors, ensuring that every student is assessed fairly and consistently. The system also reduces the resource demands associated with traditional exams, streamlining the process and making it more efficient. Ultimately, AVEA enhances the learning experience by providing immediate feedback, helping students identify areas for improvement and strengthening their clinical competencies. Through this innovation, AVEA contributes to the overall goal of preparing highly skilled medical professionals equipped to deliver high-quality patient care.

AVEA leverages advanced technologies, including speech-to-text transcription, natural language processing, and content analysis, to automate verbal exam assessments. The system relies on cloud-based infrastructure and GPU-powered servers to handle real-time transcription and evaluation. Key development resources include high-quality audio recording equipment, skilled software developers, UI/UX designers, data scientists, AI specialists, and medical academics. A user-friendly interface is designed for seamless interaction, with input from medical educators and students to ensure its practicality and effectiveness. To validate the system’s capabilities and gather user feedback, pilot programs and focus groups with educators and students will be conducted. These tests will provide insights into system performance, helping refine both the technology and user experience.
Core Functionalities:
• Real-time transcription of verbal exam responses ensures accuracy in capturing student answers.
• Natural language processing analyses responses based on criteria such as accuracy, depth, coherence, and relevance.
• AI generates immediate feedback, allowing students to understand their strengths and areas for improvement.
• The system evaluates student responses and assigns scores based on predefined standards.
• The tool provides detailed performance analytics and historical data, offering valuable insights for both students and educators.
User Experience and Performance Metrics:
AVEA is designed with simplicity and accessibility, ensuring ease of use for both students and instructors. Key performance metrics will include the system's accuracy in evaluating responses, processing speed, consistency with human graders, and user satisfaction, all of which will be monitored and refined through ongoing feedback and iteration.

Not Applicable.

AVEA introduces a novel approach to medical verbal exam assessments by integrating advanced technologies to automate and standardise the evaluation process. Unlike traditional exams, which are prone to subjectivity and examiner biases, AVEA uses speech-to-text transcription and natural language processing to provide real-time, objective assessments. By incorporating these technologies, the system ensures that every student is evaluated consistently and fairly, regardless of the examiner’s personal biases or interpretation.
Natural language processing allows AVEA to understand and assess nuanced responses, ensuring that students' answers are evaluated based on their accuracy, depth, and clarity. The AI-driven system can assess complex medical terminology and clinical reasoning, enhancing the accuracy of evaluations far beyond what a human examiner might achieve in real time.
One of AVEA's most creative aspects is its ability to provide instant feedback to students. This feature transforms the traditional exam experience, offering students the opportunity to learn from their responses immediately rather than waiting for a delayed grade. Instant feedback fosters a more dynamic learning environment and allows students to identify areas for improvement while the material is still fresh in their minds.
Additionally, automating the assessment process eliminates the logistical challenges and resource burdens associated with traditional exams, such as scheduling, manual grading, and examiner availability. By reducing administrative workload, AVEA provides a scalable solution that can be implemented across various educational institutions, offering a more efficient and effective method for conducting verbal exams at scale.
AVEA’s creative integration of AI technologies transforms the assessment process, making it more objective, efficient, and practical while improving the overall educational experience for students and educators alike.

AVEA is designed to be scalable and adaptable, ensuring that it can meet the growing demands of medical education across various institutions. Built on cloud-based infrastructure, the system can handle an increasing number of users without requiring significant additional hardware investments. This allows it to be easily deployed across multiple institutions, accommodating large cohorts of students and ensuring a consistent, high-quality assessment experience. The cloud-based setup also facilitates seamless updates and improvements to the system as technology advances, allowing AVEA to stay at the forefront of educational tools.
For sustainability, AVEA is designed with flexibility in mind. Its modular architecture ensures that it can be continuously updated with new features and improvements, such as enhanced language models or updated medical terminology, without disrupting its core functions. This adaptability makes AVEA a long-term solution that can evolve in response to changing educational and technological needs.
The system also contributes to broader educational goals of equity and inclusion. By providing standardised, unbiased assessments, AVEA ensures that students, regardless of their background or location, receive fair and equal opportunities for evaluation. This is especially important in regions with limited access to qualified examiners.
To measure its impact, AVEA will track adoption rates across different institutions, monitor user satisfaction, and evaluate improvements in student performance. By continuously gathering feedback from educators and students, AVEA will evolve to meet the shifting needs of medical education, ensuring it remains relevant and effective. This commitment to ongoing improvement and user-centric design guarantees that AVEA will continue to contribute to advancing medical education and preparing skilled healthcare professionals.

Ensures fairness and inclusivity in assessments, promotes equitable education, environmentally sustainable by reducing resource usage.
General benefits: Improves medical training quality, reduces examiner workload, optimizes resource utilization, and enhances overall healthcare outcomes.

Personal Information Collection Statement (PICS):
1. The personal data collected in this form will be used for activity-organizing, record keeping and reporting only. The collected personal data will be purged within 6 years after the event.
2. Please note that it is obligatory to provide the personal data required.
3. Your personal data collected will be kept by the LTTC and will not be transferred to outside parties.
4. You have the right to request access to and correction of information held by us about you. If you wish to access or correct your personal data, please contact our staff at lttc@eduhk.hk.
5. The University’s Privacy Policy Statement can be access at https://www.eduhk.hk/en/privacy-policy.

• I have read and agree to the competition rules and privacy policy.