{"id":12601,"date":"2026-06-20T15:52:07","date_gmt":"2026-06-20T10:22:07","guid":{"rendered":"https:\/\/eng.jfn.ac.lk\/mlsp\/?page_id=12601"},"modified":"2026-06-20T16:31:36","modified_gmt":"2026-06-20T11:01:36","slug":"speech-to-sign-healthcare","status":"publish","type":"page","link":"https:\/\/eng.jfn.ac.lk\/mlsp\/?page_id=12601","title":{"rendered":"Breaking Communication Barriers in Healthcare"},"content":{"rendered":"
Research Project · MLSP<\/span><\/p>\n

Breaking Communication Barriers in Healthcare<\/h1>\n

A personalized Speech-to-Sign Language avatar for assistive communication<\/p>\n<\/div>\n

\n
\n

Overview<\/h2>\n

Many deaf and hard-of-hearing people struggle to understand doctors who do not know sign language, and interpreters are not always available — so vital information about symptoms, medication and treatment can be misunderstood. This project develops a Personalized Speech-to-Sign Language Avatar<\/strong>: the doctor speaks naturally in Sinhala and an intelligent 3D avatar<\/strong> translates the message into Sinhala Sign Language (SSL)<\/strong> for deaf patients.<\/p>\n

Unlike systems that translate word-by-word, our approach first understands the meaning<\/em> of the message and then generates natural gestures and facial expressions — making healthcare communication more accurate, understandable and human-like.<\/p>\n<\/section>\n

\n

Methodology & Workflow<\/h2>\n
\n
\n
1<\/div>\n
\n

Speech Input<\/h4>\n

The doctor speaks naturally into a microphone; the system captures the spoken Sinhala speech.<\/p>\n<\/div>\n<\/div>\n

\n
2<\/div>\n
\n

Speech-to-Text<\/h4>\n

The recorded speech is converted into Sinhala text using speech-recognition technology.<\/p>\n<\/div>\n<\/div>\n

\n
3<\/div>\n
\n

Understanding the Message<\/h4>\n

The text is analysed to capture the meaning of the sentence rather than translating word by word.<\/p>\n<\/div>\n<\/div>\n

\n
4<\/div>\n
\n

Sign Language Translation<\/h4>\n

The sentence is mapped to Sinhala Sign Language using a sign database and gesture mappings.<\/p>\n<\/div>\n<\/div>\n

\n
5<\/div>\n
\n

Gesture Generation<\/h4>\n

Hand movements, body postures and facial expressions are generated to represent the message.<\/p>\n<\/div>\n<\/div>\n

\n
6<\/div>\n
\n

3D Avatar Animation<\/h4>\n

A 3D avatar performs the generated signs and expressions as a natural visual representation.<\/p>\n<\/div>\n<\/div>\n

\n
7<\/div>\n
\n

Visual Output<\/h4>\n

The sign-language animation is shown on screen so the deaf patient understands clearly.<\/p>\n<\/div>\n<\/div>\n

\n
8<\/div>\n
\n

System Evaluation<\/h4>\n

The system is tested and refined on translation accuracy, sign clarity and response speed.<\/p>\n<\/div>\n<\/div>\n<\/div>\n

End-to-end workflow<\/p>\n

\n
\n
1<\/div>\n

Doctor Speech<\/h4>\n<\/div>\n
\n
2<\/div>\n

Speech Recognition<\/h4>\n<\/div>\n
\n
3<\/div>\n

Meaning Analysis<\/h4>\n<\/div>\n
\n
4<\/div>\n

Sign Translation<\/h4>\n<\/div>\n
\n
5<\/div>\n

Gesture Generation<\/h4>\n<\/div>\n
\n
6<\/div>\n

3D Avatar<\/h4>\n<\/div>\n
\n
7<\/div>\n

Patient Understanding<\/h4>\n<\/div>\n<\/div>\n<\/section>\n
\n

Research Team<\/h2>\n
\n
\"Group<\/p>\n

Group Member<\/h4>\n
Undergraduate Researcher<\/div>\n
MLSP Research Group<\/div>\n<\/div>\n
\"Group<\/p>\n

Group Member<\/h4>\n
Undergraduate Researcher<\/div>\n
MLSP Research Group<\/div>\n<\/div>\n<\/div>\n<\/section>\n
← Back to Projects<\/a><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Research Project · MLSP Breaking Communication Barriers in Healthcare A personalized Speech-to-Sign Language avatar for assistive communication Overview Many deaf and hard-of-hearing people struggle to understand doctors who do not know sign language, and interpreters are not always available — so vital information about symptoms, medication and treatment can be misunderstood. This project develops a Personalized Speech-to-Sign Language Avatar: the doctor speaks naturally in Sinhala and an intelligent 3D avatar translates the message into Sinhala Sign Language (SSL) for deaf patients. Unlike systems that translate word-by-word, our approach first understands the meaning of the message and then generates natural gestures and facial expressions — making healthcare communication more accurate, understandable and human-like. Methodology & Workflow 1 Speech Input The doctor speaks naturally into a microphone; the system captures the spoken Sinhala speech. 2 Speech-to-Text The recorded speech is converted into Sinhala text using speech-recognition technology. 3 Understanding the Message The text is analysed to capture the meaning of the sentence rather than translating word by word. 4 Sign Language Translation The sentence is mapped to Sinhala Sign Language using a sign database and gesture mappings. 5 Gesture Generation Hand movements, body postures and facial expressions are generated to represent the message. 6 3D Avatar Animation A 3D avatar performs the generated signs and expressions as a natural visual representation. 7 Visual Output The sign-language animation is shown on screen so the deaf patient understands clearly. 8 System Evaluation The system is tested and refined on translation accuracy, sign clarity and response speed. End-to-end workflow 1 Doctor Speech 2 Speech Recognition 3 Meaning Analysis 4 Sign Translation 5 Gesture Generation 6 3D Avatar 7 Patient Understanding Research Team Group Member Undergraduate Researcher MLSP Research Group Group Member Undergraduate Researcher MLSP Research Group ← Back to Projects<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=\/wp\/v2\/pages\/12601"}],"collection":[{"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12601"}],"version-history":[{"count":1,"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=\/wp\/v2\/pages\/12601\/revisions"}],"predecessor-version":[{"id":12614,"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=\/wp\/v2\/pages\/12601\/revisions\/12614"}],"wp:attachment":[{"href":"https:\/\/eng.jfn.ac.lk\/mlsp\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12601"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}