Open Category Submission

395 – AICan – AI for Cantonese Education

This project aims to develop an AI-powered educational platform to assist in teaching and learning Cantonese. The website integrates advanced technologies to create a seamless and interactive learning experience, fostering a collaborative student-teacher-AI loop.

Key Innovations:
1. Multimodal Interactive Learning: AI-driven real-time voice conversations help students improve their pronunciation and fluency. Also, users can upload images, to which AI will generate Cantonese vocabulary and explanations related.
2. Multilingual Translation: Real-time translation between Cantonese and other languages.
3. Sentence Conversion and Feedback: Users can input sentences in their native language, and the AI will provide the Cantonese equivalent, along with pronunciation guidance and accuracy scoring.
4. Teacher Integration: The platform will support teachers by providing tools to monitor student progress, assign tasks, and offer personalized feedback, ensuring AI complements human instruction.

Design Concepts:
The core objective of this project is to address the challenges faced by students learning Cantonese, particularly the lack of a conversational environment and the limited support for Cantonese in existing large language models (LLMs), such as grammar, vocabulary, and pronunciation. By leveraging AI-powered voice conversations, this platform aims to assist students and teachers in the classroom while also serving as an independent training tool. The design principles focus on engaging students through gamification, enhancing usability, fostering continuous learning, and ensuring a user-friendly experience.

Technical Principles:
The project leverages state-of-the-art LLMs, speech recognition and synthesis models tailored for Cantonese. It will use cloud-based infrastructure for scalability and real-time processing.

Potential Impact:
This project fills the gap for Cantonese learners, supports classroom teaching, promotes self-study, and contributes to education equity by making practice accessible and engaging. Beyond learning, it helps reach more people, blend cultures, and create fresh contents, keeping Cantonese language and culture stay vibrant and evolve—with potentials to extend to other low-resource languages and cultural heritage.

• Pitch-Deck_AICan.pdf

1. Inspiration:
As Cantonese learners, we have always dreamed of speaking fluently with locals to better integrate into Hong Kong's vibrant community and campus life. Yet despite years of effort—taking in-person classes, using language apps, even experimenting with AI tools—we still have not reached our learning goals.
For example, we enrolled in offline courses, but teachers couldn't practice with me beyond weekly sessions, leaving us without consistent training or instant feedback. we tried apps like Duolingo, but they just recycled the same basic phrases without simulating real conversations. Later, we tested OpenAI's chatbot, but its Cantonese support was poor—incorrect grammar and unnatural pronunciation made it unusable.
Our friends share similar frustrations. No matter what methods we try, we always hit the same roadblocks: lack of high-frequency practice in authentic contexts and no real-time correction.

2. Hypothesis:
With the development of AI language models and audio processing techniques like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS), there are increasing demands of involving more advanced and intelligent solutions to improve the efficiency and popularity of language education. The success of an AI-assisted language education system requires deep interaction among students, teachers and AI systems. Therefore, we gather our team by inviting professional researchers in language education and linguistics, in AI audio and language models and obtaining on-site class resources and real-time interaction between students and teachers by connecting with language centers at universities.
Moreover, multilingual learners are extremely common worldwide, as evidenced by the success of language learning software like Duolingo. Our student-teacher-AI loop system can also be extended to other languages, enabling this application to reach more potential users.

For feasibility:
1. We will leverage state-of-the-art (SOTA) audio processing techniques, such as ASR, dialogue understanding with LLMs, and TTS, and advance these multi-stage modules into a one-stage streaming approach. We also need to ensure that this chat-based system does not output toxic content.
2. The resources for model training that are needed include high-quality Cantonese audio-text pair datasets, computing power, and platform maintenance.
3. We have partially validated our market demand based on a questionnaire and customer interviews with Chinese mainland users and will expand to more groups from different countries. The results are illustrated in the supplementary materials. In addition to student users, we also plan to investigate the detailed demands of teacher users by auditing their on-site Cantonese classes and collecting their feedback on existing AI-based education applications.

For functionality:
1. Our solution delivers core functionalities that include not only ChatGPT-like functions for Cantonese, such as text-based conversation and real-time voice chat, but also go beyond GPT, as it is specially designed for linguistic educational use. These features include professionally designed courses, grammar checking, and advanced pronunciation grading with detailed error analysis.
2. To enhance user experience, we combine an intuitive user interface with gamification elements like leaderboards and daily check-in rewards. We also aim to ensure alignment with teacher-led instruction through topic-specific exercises (e.g., numbers or restaurant ordering). Teachers guide students effectively using the platform, creating a cohesive learning experience.
3. Effectiveness is measured through quantifiable progress tracking, such as visualizations of pronunciation accuracy improvements over time. Dr. Lee (on our team) is also planning to conduct a controlled study comparing our AI-assisted classes with traditional instruction in university settings, providing empirical evidence of the solution's educational impact.

Not applicable

Our solution innovates across interactivity, personalization, accessibility, and integration, by incorporating diverse learning pathways based on insights from surveys and interviews, addressing students' pain points and aligning with their preferred learning methods to ensure a tailored and effective approach.

1. Steep Learning Curve? Can’t understand? -> Multilingual Translation: Real-time translation between Cantonese and other languages enhances learners' understanding of their native language, bridging communication gaps and fostering deeper comprehension.
2. Lacking Practice? -> Voice-Based Interaction: Our AI-driven real-time voice conversations provide an immersive way for students to practice and improve pronunciation and fluency, making language learning more engaging and effective.
3. Don’t Know Your Mistake? -> Real-Time Feedback: Based on user input—whether voice or text—the AI provides instant feedback on pronunciation, grammar, and sentence structure, helping learners correct mistakes and improve their skills instantly.
4. Hybrid Learning Approaches!
4.1 Sentence Conversion and Feedback: Besides having conversations with AI, we also allow users to input sentences in their native language and receive Cantonese equivalents, pronunciation guidance, and accuracy scoring, we offer a personalized and interactive learning experience.
4.2 Image-Based Learning: Users can upload images, and the AI generates related Cantonese vocabulary and explanations, creating a visual and context-rich learning environment.
4.3 Teacher Integration: The platform supports teachers with tools to monitor progress, assign tasks, and provide personalized feedback, ensuring AI complements human instruction for a balanced and effective learning approach.

To ensure our conversational AI system for Cantonese and other low-resource language education is both scalable and sustainable, we’ve developed a comprehensive approach that balances technical innovation, environmental responsibility, and user-centered design.

Scalability Strategies:
1. Modular, Cloud-Based Design: We’ve built the system on a modular architecture, allowing individual components like ASR, NLP, and TTS to scale independently. By leveraging cloud infrastructure and distributed databases, we can handle growing user numbers without sacrificing performance.
2. Efficient Model Training: We use transfer learning and few-shot learning to adapt high-resource language models for low-resource languages. This reduces computational costs and speeds up deployment. Additionally, edge AI deployment ensures accessibility in areas with limited internet connectivity.
3. Data Expansion Techniques: To address data scarcity, we employ synthetic data generation, back-translation, and multilingual embeddings. We also engage local communities and educators to crowdsource data, ensuring the system remains adaptable to diverse languages and dialects.
4. Streamlined Processing: By transitioning to end-to-end speech-to-speech architectures, we’ve simplified the processing pipeline, enabling faster, real-time interactions and reducing latency.

Sustainability Strategies:
1. Environmental Responsibility: Our cloud infrastructure prioritizes energy-efficient computing, and we use optimized inference techniques to minimize energy consumption and carbon emissions.
2. User-Centric Design: The system adapts to individual learners through personalized learning paths, which evolve based on user progress and feedback. Gamification and interactive features keep users engaged and motivated over time.
3. Community Collaboration: We work closely with educators, linguists, and local communities to ensure the content is culturally relevant and impactful. Open-source initiatives and partnerships with academic institutions, like the language center at HKUST, further promote long-term sustainability and knowledge sharing.
4. Inclusive Accessibility: The platform is designed to run on a wide range of devices, from high-end desktops to low-cost smartphones, ensuring accessibility for users across different socioeconomic backgrounds.

Our solution addresses the growing digital language divide by empowering speakers of low-resource languages through accessible, AI-driven education. While most AI tools cater to mainstream languages, we focus on preserving linguistic diversity and promoting inclusion for marginalized communities. By integrating conversational AI into language learning, we aim to bridge educational gaps, foster cultural preservation, and create equitable opportunities for all.

Key Contributions to Equity and Inclusion:
1. Closing the Language Gap: Our system provides high-quality language learning tools for underserved communities. Using human-in-the-loop data collection and multilingual AI, we ensure that even the most underrepresented languages are supported, opening doors to better education and economic opportunities.
2. Cultural Preservation: Many low-resource languages face extinction due to lack of digital representation. Our AI models are designed to recognize and generate these languages accurately, working closely with native speakers and linguists to preserve cultural heritage.
3. Accessible and Personalized Learning: We prioritize inclusivity by offering adaptive learning paths and speech-based interfaces. This ensures accessibility for users with disabilities and those in remote areas with limited resources.

Measuring Impact and Staying Responsive:
To evaluate our impact, we track:
1. Language Retention: How well learners maintain their native language skills, measured through longitudinal studies tracking intergenerational language use and fluency levels over time.
2. Learning Outcomes: Improvements in proficiency and engagement rates, assessed via AI-driven progress tracking, test scores, and time spent actively using the platform.
3. Community Involvement: Participation of local communities in data collection and feedback, ensuring that the system reflects their linguistic nuances and cultural values.
4. Accessibility Metrics: Usage rates among diverse socioeconomic and demographic groups, with a focus on reaching underserved populations, including rural communities and individuals with disabilities.

Personal Information Collection Statement (PICS):
1. The personal data collected in this form will be used for activity-organizing, record keeping and reporting only. The collected personal data will be purged within 6 years after the event.
2. Please note that it is obligatory to provide the personal data required.
3. Your personal data collected will be kept by the LTTC and will not be transferred to outside parties.
4. You have the right to request access to and correction of information held by us about you. If you wish to access or correct your personal data, please contact our staff at lttc@eduhk.hk.
5. The University’s Privacy Policy Statement can be access at https://www.eduhk.hk/en/privacy-policy.

• I have read and agree to the competition rules and privacy policy.