Breaking Communication Barriers in Healthcare
A personalized Speech-to-Sign Language avatar for assistive communication
Overview
Many deaf and hard-of-hearing people struggle to understand doctors who do not know sign language, and interpreters are not always available — so vital information about symptoms, medication and treatment can be misunderstood. This project develops a Personalized Speech-to-Sign Language Avatar: the doctor speaks naturally in Sinhala and an intelligent 3D avatar translates the message into Sinhala Sign Language (SSL) for deaf patients.
Unlike systems that translate word-by-word, our approach first understands the meaning of the message and then generates natural gestures and facial expressions — making healthcare communication more accurate, understandable and human-like.
Methodology & Workflow
Speech Input
The doctor speaks naturally into a microphone; the system captures the spoken Sinhala speech.
Speech-to-Text
The recorded speech is converted into Sinhala text using speech-recognition technology.
Understanding the Message
The text is analysed to capture the meaning of the sentence rather than translating word by word.
Sign Language Translation
The sentence is mapped to Sinhala Sign Language using a sign database and gesture mappings.
Gesture Generation
Hand movements, body postures and facial expressions are generated to represent the message.
3D Avatar Animation
A 3D avatar performs the generated signs and expressions as a natural visual representation.
Visual Output
The sign-language animation is shown on screen so the deaf patient understands clearly.
System Evaluation
The system is tested and refined on translation accuracy, sign clarity and response speed.
End-to-end workflow
Doctor Speech
Speech Recognition
Meaning Analysis
Sign Translation
Gesture Generation
3D Avatar
Patient Understanding
Research Team
Group Member
