Open Category Submission

323 – Enhance social and emotional learning of adolescents with autism spectrum disorder through a gamified interactive virtual community of generative agents (ESAIAS)

This project introduces a virtual simulation system, ESAIAS, to transform social and emotional learning (SEL) for adolescents with autism spectrum disorder (ASD) in Hong Kong's integrated education setting. ESAIAS leverages generative artificial intelligence (GenAI) to address the key limitations of traditional methods, such as human-led role-playing. Unlike conventional approaches that require intensive therapist involvement, physical presence, and fixed scenarios, ESAIAS offers scalable, accessible, and adaptive training within a virtual simulation of a school community. Students interact with GenAI agents to complete tasks involving collaboration, conflict resolution, and emotional regulation.

The technical framework consists of (i) a front end that provides stylised visuals and natural interactions, (ii) a back end that hosts GenAI agents with pre-configured personas, dynamic memory, and role-playing algorithms, and (iii) a set of bespoke tools for system monitoring, agent configuration, and scenario design.

The impact of ESAIAS lies in its ability to overcome the limitations of traditional SEL methods for adolescents with ASD. While human-facilitated approaches are resource-intensive and location-dependent, ESAIAS offers unmatched scalability and accessibility through GenAI technology. It allows an unlimited number of students simultaneous use and enhances accessibility via digital delivery to schools and homes. Additionally, the adaptability of ESAIAS surpasses traditional methods; GenAI agents can integrate an endless number of social nuances, personalise their interactions based on individual ability and skills, and quickly respond to newly arising needs.

Early feedback from special education teachers, counsellors, psychologists, and psychiatrists suggests that students are likely to engage more deeply in SEL with GenAI agents, since the absence of others' judgment encourages experimentation and reduces anxiety. By providing an immersive, safe, replicable, and personalised learning experience, this innovative approach aims to democratise access to high-quality SEL for adolescents with ASD, particularly in under-resourced settings. It also lays the groundwork for GenAI-enhanced interventions across neurodiverse learners, ultimately fostering inclusive communities where adolescents with special educational needs (SEN) can thrive.

Adolescents with ASD often face profound challenges in navigating social interactions, such as interpreting non-verbal cues, managing emotional reciprocity, and adapting to group dynamics, leading to isolation, academic underperformance, and mental health struggles. Traditional SEL approaches, such as therapist-led role-playing, social stories, or group workshops, rely heavily on human facilitation, rigid scripts, and controlled environments. While beneficial, these methods lack accessibility, scalability, adaptability, and ecological validity.

This project was inspired by two key insights: the transformative potential of GenAI to simulate nuanced social scenarios and the pressing need for adaptive SEL solutions tailored to adolescents with ASD in Hong Kong's integrated education setting. The hypothesis driving this project is that a virtual simulation using GenAI agents, designed collaboratively with professionals and grounded in task-based contextual learning, will enhance SEL outcomes more effectively than conventional methods. This belief is anchored in four pillars:

(1) Proven Efficacy of Role-Playing: Extensive research validates role-playing as an effective SEL tool for children with ASD. However, its reliance on human facilitators limits accessibility and scalability. Integrating GenAI can address these shortcomings.
(2) Empirical Evidence of Task-Based Learning: Our team's school-based studies on structured social tasks and contextual learning demonstrated measurable improvements in social and emotional skills among neurodiverse learners.
(3) Advancements in GenAI Technologies: Recent developments in GenAI, particularly in context-aware dialogue generation and modelling socially nuanced responses, enable agents to interpret and respond to complex social cues in ecologically valid ways.
(4) Co-Design with Professionals: By collaborating with special education teachers, counsellors, psychologists, and psychiatrists, we have iteratively designed virtual scenarios that encompass relevant social tasks reflective of real-world challenges.

By bringing these elements together, this project is expected to succeed by overcoming systemic barriers in SEL for adolescents with ASD.

N/A

The technical framework of ESAIAS has three main components designed to provide accessible, scalable, and adaptive SEL for adolescents with ASD.

The front end, developed using Unity, features stylised visuals and facilitates natural interactions. It enables students to navigate virtual scenarios and freely engage in voice-based conversations with GenAI agents to complete social interaction tasks by following visual hints. The front end connects to the back end through a representational state transfer (REST) architecture, the industry standard in networked application development due to its scalability, flexibility, and security.

The back end, implemented in Python, endows GenAI agents with a deep understanding of role-playing through carefully designed prompts and algorithms that govern memory and behaviour, restricting the agents to role-related and ecologically valid responses. The back end uses state-of-the-art open-source LLMs to provide the agents with foundational reasoning and dialogue capabilities.

A set of bespoke tools has been developed to facilitate the intuitive design of GenAI agents, social tasks, and virtual scenarios. Professionals can create tailored SEL content using a drag-and-drop interface, pre-built behavioural templates, and natural language prompts, regardless of their technical expertise. This customised content can be saved as configuration files in JSON format, which the back-end system interprets.

In addition to the three technical components that have already been implemented, ESAIAS plans to include five virtual scenarios, each corresponding to tasks that focus on the five core competencies of SEL: self-awareness, self-management, social awareness, relationship skills, and responsible decision-making. After nine months in development, we have successfully implemented two of the five planned scenarios, and we expect to complete all five scenarios within 12 months of the project's kick-off.

To thoroughly evaluate the feasibility and effectiveness of ESAIAS on SEL, a pilot trial followed by a randomised controlled trial (RCT) is planned. This evaluation will utilise well-validated questionnaires and system logs to gather comprehensive data.

ESAIAS represents a groundbreaking fusion of GenAI, gamified simulation, and co-designed approach with professionals, redefining SEL for adolescents with ASD. Unlike conventional methods constrained by static scripts or human resource limitations, ESAIAS innovates through three core aspects: flexibility in deployment, scalability via bespoke tools, and adaptability through personalisation.

Flexibility in Deployment: ESAIAS is engineered for universal accessibility. It can operate on local machines equipped with modern neural processing units (NPUs) for offline use or scale seamlessly via cloud infrastructure to support countless simultaneous users. This dual deployment capability breaks geographical and financial barriers inherent in traditional SEL programmes.

Scalability via Bespoke Tools: Another innovation of ESAIAS lies in its suite of user-friendly, visualised, and wizard-based tools for designing GenAI agents, social tasks, and virtual scenarios. Professionals, such as special education teachers and counsellors, can craft tailored SEL content using a drag-and-drop interface, prebuilt behavioural templates, and natural language prompts, regardless of their technical expertise. These tools simplify complex GenAI workflows, empowering us to rapidly expand SEL content with professional inputs, so that the content stays relevant to evolving student needs.

These innovations collectively address the rigidity and resource dependency of traditional SEL. By merging intuitive design tools with GenAI agents' fast-evolving capabilities, ESAIAS transcends the "one-size-fits-all" approach, offering an accessible and scalable yet deeply personalised learning experience. Its innovation and creativity lie deeply in transforming advanced AI technologies into an invisible and empowering force. This positions ESAIAS not only as a pioneering work for SEL among adolescents with ASD but also as a powerful and promising learning platform for neurodiverse learners.

Adaptability through Personalisation: ESAIAS leverages GenAI to deliver unmatched personalisation. Each interaction dynamically adapts based on the student's progress; GenAI agents can adjust dialogue complexity, emotional tones, and task difficulty in real time, guided by memory algorithms that track individual strengths and challenges. This responsiveness mimics the ideal one-on-one coaching experience, which is often impossible in group-based human interventions.

ESAIAS is designed for scalability through a highly elastic architecture that supports both local deployment (via NPU-enabled devices) and cloud-based expansion. Cloud integration allows dynamic resource allocation, enabling the system to handle countless simultaneous users by automatically scaling server instances during spikes in demand. This architecture also addresses a potential bottleneck in latency from LLM processing and enhances the system's environmental sustainability.

For long-term engagement, ESAIAS leverages personalised learning pathways, where GenAI agents dynamically adjust task difficulty and social complexity based on individual progress. Gamified elements, such as achievement badges, progress dashboards, and unlockable scenarios, motivate consistent use. The system's bespoken tools allow easy updates to the virtual scenarios, informed by educator feedback and co-designed with professionals, to ensure content remains fresh and relevant.

The bespoken tools also enhance the system's adaptation to evolving needs. These design tools allow professionals, such as special education teachers and counsellors, to craft tailored SEL content using a drag-and-drop interface, prebuilt behavioural templates, and natural language prompts, regardless of their level of technical expertise. This democratises SEL content creation, enabling rapid iteration to reflect shifts in learning needs, emerging social challenges, or individual student goals. Furthermore, the modular server design allows seamless integration of future AI advancements (e.g., emotion recognition via multimodal LLMs), ensuring the platform evolves alongside pedagogical and technological trends.

ESAIAS directly tackles social challenges faced by adolescents with ASD in Hong Kong's integrated education setting. Traditional SEL methods often rely on resource-intensive, human-led role-playing that is not easily accessible or adaptable to individual needs. ESAIAS leverages GenAI to produce a scalable, accessible, and adaptive virtual simulation. It enhances the lives of its primary beneficiaries by offering immersive, interactive, and personalised learning experiences where students can hone essential social skills, such as collaboration, conflict resolution, and emotional regulation, in a safe and judgment-free environment.

By enabling students to interact with GenAI agents that can generate endless social nuances and tailor interactions to individual abilities, ESAIAS addresses the specific social issue of limited access to effective SEL for adolescents with ASD. It reduces anxiety and encourages experimentation, leading to deeper engagement. This aligns with broader equity and inclusion social goals by democratising access to high-quality SEL resources, especially in under-resourced settings. ESAIAS fosters inclusive communities by laying the groundwork for GenAI-enhanced interventions across neurodiverse learners.

We will employ quantitative and qualitative metrics to measure the project's social impact. Quantitatively, we will use standardised assessments to evaluate improvements in social-emotional competencies before and after using ESAIAS. Engagement metrics such as usage frequency, session duration, and task completion rates within the system will also be tracked. Qualitatively, we will gather feedback from teachers and parents to assess changes in social behaviour and emotional well-being. Additionally, we will monitor the impact on scalability and accessibility by tracking the number of users.

Ensuring responsiveness to the community's evolving needs is integral to our approach. ESAIAS includes feedback mechanisms that allow users to report experiences and suggest improvements. Based on this feedback, our bespoke tools enable swift updates to GenAI agent configurations and scenario designs. Regular consultations with professionals and the ASD community will inform ongoing development.

Personal Information Collection Statement (PICS):
1. The personal data collected in this form will be used for activity-organizing, record keeping and reporting only. The collected personal data will be purged within 6 years after the event.
2. Please note that it is obligatory to provide the personal data required.
3. Your personal data collected will be kept by the LTTC and will not be transferred to outside parties.
4. You have the right to request access to and correction of information held by us about you. If you wish to access or correct your personal data, please contact our staff at lttc@eduhk.hk.
5. The University’s Privacy Policy Statement can be access at https://www.eduhk.hk/en/privacy-policy.

• I have read and agree to the competition rules and privacy policy.