Health Technologies

AI could advance inclusivity by interpreting sign language in real time

Researchers in the US have conducted a first-of-its-kind study focused on recognising American Sign Language alphabet gestures using computer vision.

The research could play a key role in breaking down communication barriers and ensuring more inclusive interactions.

The researchers developed a custom dataset of 29,820 static images of American Sign Language hand gestures.

Using MediaPipe, each image was annotated with 21 key landmarks on the hand, providing detailed spatial information about its structure and position.

These annotations played a critical role in enhancing the precision of YOLOv8, the deep learning model the researchers trained, by allowing it to better detect subtle differences in hand gestures.

Results of the study reveal that by leveraging this detailed hand pose information, the model achieved a more refined detection process, accurately capturing the complex structure of American Sign Language gestures.

Combining MediaPipe for hand movement tracking with YOLOv8 for training, resulted in a powerful system for recognising American Sign Language alphabet gestures with high accuracy.

First author Bader Alsharif is a Ph.D. candidate in the Florida Atlantic University (FAU) Department of Electrical Engineering and Computer Science.

The researcher said: “Combining MediaPipe and YOLOv8, along with fine-tuning hyperparameters for the best accuracy, represents a groundbreaking and innovative approach.

“This method hasn’t been explored in previous research, making it a new and promising direction for future advancements.”

Findings show that the model performed with an accuracy of 98 per cent, the ability to correctly identify gestures (recall) at 98 per cent, and an overall performance score (F1 score) of 99 per cent.

It also achieved a mean Average Precision (mAP) of 98 per cent and a more detailed mAP50-95 score of 93 per cent, highlighting its strong reliability and precision in recognising American Sign Language gestures.

Alshari said: “Results from our research demonstrate our model’s ability to accurately detect and classify American Sign Language gestures with very few errors.

“Importantly, findings from this study emphasise not only the robustness of the system but also its potential to be used in practical, real-time applications to enable more intuitive human-computer interaction.”

The successful integration of landmark annotations from MediaPipe into the YOLOv8 training process significantly improved both bounding box accuracy and gesture classification, allowing the model to capture subtle variations in hand poses.

This two-step approach of landmark tracking and object detection proved essential in ensuring the system’s high accuracy and efficiency in real-world scenarios.

The model’s ability to maintain high recognition rates even under varying hand positions and gestures highlights its strength and adaptability in diverse operational settings.

Mohammad Ilyas, Ph.D., is co-author and a professor in the FAU Department of Electrical Engineering and Computer Science.

Ilyas said: “Our research demonstrates the potential of combining advanced object detection algorithms with landmark tracking for real-time gesture recognition, offering a reliable solution for American Sign Language interpretation.

“The success of this model is largely due to the careful integration of transfer learning, meticulous dataset creation, and precise tuning of hyperparameters.

“This combination has led to the development of a highly accurate and reliable system for recognising American Sign Language gestures, representing a major milestone in the field of assistive technology.”

Future efforts will focus on expanding the dataset to include a wider range of hand shapes and gestures to improve the model’s ability to differentiate between gestures that may appear visually similar, thus further enhancing recognition accuracy.

Additionally, optimising the model for deployment on edge devices will be a priority, ensuring that it retains its real-time performance in resource-constrained environments.

Stella Batalama, Ph.D., dean, FAU College of Engineering and Computer Science, said: “By improving American Sign Language recognition, this work contributes to creating tools that can enhance communication for the deaf and hard-of-hearing community.

“The model’s ability to reliably interpret gestures opens the door to more inclusive solutions that support accessibility, making daily interactions – whether in education, health care, or social settings – more seamless and effective for individuals who rely on sign language.

“This progress holds great promise for fostering a more inclusive society where communication barriers are reduced.”

Image: Florida Atlantic University

Avatar

admin

About Author

You may also like

Health Technologies

Accelerating Strategies Around Internet of Medical Things Devices

  • December 22, 2022
IoMT Device Integration with the Electronic Health Record Is Growing By their nature, IoMT devices are integrated into healthcare organizations’
Health Technologies

3 Health Tech Trends to Watch in 2023

Highmark Health also uses network access control technology to ensure computers are registered and allowed to join the network. The