Higher Education Category Submission

408 – Facial Expression Recognition, Sign Language and English Bilingual Learning Software

Project Description
This project, developed by students from IVE (Sha Tin) studying Higher Diploma in Software Engineering, focuses on creating an innovative tool to support bilingual education and emotional development for preschool and kindergarten children, including those with Special Educational Needs (SEN). Inspired by research from Prof. Gladys Tang and Dr. Wingchee So at CUHK, it emphasizes inclusive communication through English, sign language, and emotional learning.

The key innovation is an AI-driven platform that integrates facial expression recognition, sign language recognition, and phonics education. Using technologies like Google AI Edge (MediaPipe), CNNX models, and a self-trained YOLOVI model, the tool offers real-time emotion analysis, hand-tracking for sign language, and speech feedback for pronunciation. It addresses gaps in engaging resources, parental support, and educator tools, particularly in Hong Kong, where sign language learning tools are limited.

Designed with a user-centered approach, the project targets special education teachers, kindergarten teachers, community workers, and parents. It aims to improve teaching effectiveness and communication skills for children with and without SEN, tackling challenges like social misconceptions and accessibility for low-income families. The system uses a three-tier architecture—mobile/web interfaces, server logic, and a SQL data tier—for a smooth, scalable experience.

Currently in the implementation phase (December 2024 - February 2025), the project has completed requirements analysis and system design. Its potential impact lies in providing an affordable, inclusive, and comprehensive learning tool that bridges educational gaps and fosters emotional and communication skills. With testing planned for March 2025 and deployment in April 2025, this future-ready solution could transform early education by leveraging AI technology and promoting inclusivity.

Problem Identification and Relevance in Education
The idea for this project came from noticing key problems in early education, especially for children with Special Educational Needs (SEN) in Hong Kong. We were inspired by research from Prof. Gladys Tang and Dr. Wingchee So at CUHK, which showed the need for tools that support bilingual learning (English and sign language) and emotional growth. We saw that Hong Kong lacks fun, easy-to-use resources for learning sign language. Many people wrongly believe sign language harms speech development. Busy parents find it hard to help with bilingual education, and teachers lack proper tools, especially for autistic children who struggle to communicate.

These issues led us to create a solution: an AI-based tool that combines facial expression recognition, sign language recognition, and phonics learning. Our hypothesis is that this tool can improve learning for preschool and kindergarten children, with or without SEN, by making education more complete and engaging. It addresses the shortage of enjoyable, affordable resources and supports communication skills for all.

We think it will work because we listened to teachers and parents during our planning in September 2024 to understand their needs. Using smart technology, like Google AI Edge and our own trained models, ensures the tool is accurate and can grow. It is designed simply for teachers, parents, and children to use easily. This project matters because it makes education fairer and more inclusive, especially for children from poor families or with SEN. It fits with the modern push for learning that uses technology and suits each child’s needs. Its success depends on providing a helpful, accessible way to improve learning and teaching.

2a. Feasibility and Functionality
To implement our solution, we will use specific technologies like Google AI Edge (MediaPipe Face and Hand Landmarkers) for real-time facial expression and sign language recognition, and a self-trained YOLOVI model for accuracy. AI speech recognition will support phonics learning with pronunciation feedback. We will develop the tool using React Native (Expo) for mobile apps (Android and iOS) and a web platform, ensuring it works across devices. A three-tier architecture—client, server, and MySQL database—will manage user data and analytics.

Resources needed include computers, software licenses, and access to datasets like Kaggle’s ASL alphabet for training models. We will collaborate with educators and parents for feedback and testing. To validate market demand, we will survey special education teachers, kindergarten teachers, and parents during the testing phase (March 2025) to confirm the tool meets their needs.

The core functionalities are: 1) facial expression recognition to teach emotions, 2) sign language recognition with interactive lessons, and 3) AI-powered phonics for bilingual learning. To ensure a positive user experience, we will design a simple, colorful interface based on wireframes created in October-November 2024, and gather usability feedback during testing. Features like real-time feedback and engaging activities will keep users interested.

Performance metrics include: 1) accuracy of recognition (targeting 90% for facial expressions and sign language), 2) user satisfaction (measured by surveys, aiming for 80% positive responses), and 3) engagement time (tracking how long children use the tool, targeting 15-20 minutes per session). These metrics will be evaluated during the testing phase (March 2025) to ensure the tool works well and helps teachers, parents, and children effectively.

2b. Technical Implementation and Performance
The project follows a three-tier functional architecture: the presentation tier (mobile apps and web interfaces developed with React Native), the logic tier (server managing AI models and business logic), and the data tier (Firebase storing user profiles, content, and analytics). The technical workflow begins with users accessing the platform, where inputs like facial expressions, hand gestures, and speech are captured. These are processed by the server using AI models, delivering real-time feedback to the user.

Innovative features include facial expression recognition, sign language recognition, and phonics integration. For facial expression recognition, we use Google AI Edge’s MediaPipe Face Landmarker to analyze emotions. Sign language recognition employs MediaPipe Hand Landmarker and the Kaggle ASL alphabet dataset for gesture detection. Phonics integration uses AI speech recognition for pronunciation feedback. Implementation involves training models with datasets, coding features in React Native, and integrating them into the server. The design and development timeline started with requirements analysis (September 2024), followed by system design (October-November 2024). Implementation is underway (December 2024 - February 2025), with testing planned for March 2025 and deployment in April 2025.

Performance metrics include: 1) recognition accuracy (targeting 90% for expressions and gestures), 2) response time (under 2 seconds for real-time feedback), and 3) system uptime (aiming for 99% availability). These will be assessed during testing.

Function Point and Technical Application
Function Point Technical Application Progress
Facial/Sign Recognition Google AI Edge (MediaPipe Face/Hand Landmarker) Fininsed
Phonics Feedback AI Speech Recognition Development in progress
Frontend Interface React Native (Expo) Development in progress
Emotion Analysis CNNX Model Zoo - Emotion FERPlus Fininsed
Gesture Learning Modules Kaggle Dataset Fininsed
Real-Time Data Processing Self-Trained YOLOV11 Model Fininsed

These technologies ensure precise recognition, interactive learning, and a smooth interface. They support the project’s aim of enhancing communication and education, with ongoing progress toward a fully functional system by April 2025.

3. Innovation and Creativity
Our project provides a new and creative answer to the problem of missing fun and inclusive learning tools for bilingual education and emotional growth in Hong Kong, especially for children with Special Educational Needs (SEN). Unlike usual methods that focus on just one part of learning, our tool brings together facial expression recognition, sign language recognition, and phonics teaching in one AI-based platform. This fresh combination tackles several issues: the lack of enjoyable sign language resources, wrong ideas about its effect on speech, and the need for tools to teach emotions.

The innovation comes from using smart technologies like Google AI Edge (MediaPipe) and our own YOLOV11 model to give instant feedback on emotions, hand signs, and speaking. For example, children can practice sign language with interactive lessons while the tool shows their feelings on the screen, making learning lively and fun. This differs from typical teaching tools by mixing sight, sound, and movement, which helps SEN children, like those with autism who find communication hard.

Creativity is clear in how we designed it with input from teachers and parents to meet their real needs. The bright, easy-to-use design keeps young children interested, while features like AI phonics support busy parents and teachers with few resources. By helping both SEN and regular preschoolers, it makes learning fair for all—a new idea for early education tools.

These features make the tool work better by being easy to use, low-cost, and enjoyable. It fills gaps in phonics for poorer families and aids teachers with limited tools. Combining advanced AI with a focus on emotions and bilingual skills makes it both useful and forward-looking, standing out as a special fix for ongoing education problems.

4. Scalability and Sustainability
To ensure our solution scales with growing user demand, we will use a flexible system design supported by Firebase. The three-tier architecture—mobile/web interfaces, server, and Firebase database—can manage more users by leveraging Firebase’s cloud-based scalability. Firebase allows us to expand storage and processing power easily as demand increases. During the testing phase (March 2025), we will track performance to find bottlenecks, such as slow responses, and address them by optimizing Firebase queries or enhancing server efficiency. Post-deployment updates (April 2025 onward) will keep the system smooth as more teachers, parents, and children use it.

For environmental sustainability, Firebase’s efficient cloud infrastructure reduces energy use compared to physical servers. Our digital tool, built with React Native for Android and iOS, cuts down on paper waste and device needs, lowering its eco-footprint. To maintain long-term user engagement, we will use Firebase to push new content—like extra sign language lessons or phonics games—based on user feedback collected after launch. Fun, interactive features with real-time feedback will keep children interested.

To adapt to evolving needs, Firebase’s real-time database will help us update the tool quickly. We will gather input from educators and parents via surveys and adjust features, such as adding new languages or tools for older children. Firebase supports training AI models with fresh data, like new gestures or emotions, ensuring the tool stays relevant. By using affordable Firebase services and offering ongoing support, the solution can grow sustainably, reaching more schools and families while remaining cost-effective, eco-friendly, and useful over time.

5. Social Impact and Responsibility
Our solution tackles key social issues in Hong Kong, such as the lack of inclusive education tools and limited support for children with Special Educational Needs (SEN). By offering an AI-based platform with facial expression recognition, sign language recognition, and phonics, it helps preschool and kindergarten children—especially those with SEN, like autism—improve communication and emotional skills. This benefits primary users: children, special education teachers, kindergarten teachers, and busy parents. It provides affordable, engaging resources, addressing gaps in phonics for low-income families and teaching support for educators. This aligns with broader goals of equity and inclusion by making quality education accessible to all, regardless of income or ability.

The tool enhances lives by breaking down communication barriers, correcting misconceptions about sign language, and fostering emotional growth. For example, autistic children gain better ways to express themselves, while teachers get practical tools to improve lessons. Parents, often stretched for time, can support bilingual learning easily. This promotes fair access to education and builds a more inclusive community.

Personal Information Collection Statement (PICS):
1. The personal data collected in this form will be used for activity-organizing, record keeping and reporting only. The collected personal data will be purged within 6 years after the event.
2. Please note that it is obligatory to provide the personal data required.
3. Your personal data collected will be kept by the LTTC and will not be transferred to outside parties.
4. You have the right to request access to and correction of information held by us about you. If you wish to access or correct your personal data, please contact our staff at lttc@eduhk.hk.
5. The University’s Privacy Policy Statement can be access at https://www.eduhk.hk/en/privacy-policy.

• I have read and agree to the competition rules and privacy policy.