Comparison of Deep Learning Architectures for Sign Language Recognition: From Pixel-based to Holistic Keypoints on Web-based Applications

Abstract

This research aims to propose and evaluate the performance of deep learning models for the classification of 50 sign language glosses (WLASL-50) to enhance communication efficiency between hearing-impaired individuals and the general public. The comparative study investigates three distinct architectures: 1) a hybrid Convolutional Neural Network and Long Short-Term Memory model (YOLO-LRCN) leveraging raw pixel data, 2) a hand landmark-based model (MediaPipe Hands) integrated with LSTM, and 3) a holistic skeletal landmark model (MediaPipe Holistic) that fuses hand and upper-body pose coordinates with LSTM. Experimental results demonstrate that the holistic-based approach achieves superior performance, significantly increasing the Top-10 Accuracy from 45.98% in the baseline pixel-based model (YOLO-LRCN) to 81.17%. Furthermore, the holistic approach achieved a substantial Top-1 Accuracy of 44.84%. Beyond accuracy enhancements, the coordinate-based architectures exhibit a significant reduction in model complexity, featuring parameters approximately ten times fewer than those of pixel-based models (approx. 850 KB). This efficiency results in high-speed processing, with an average inference time of approximately 95 ms per prediction, and minimal computational resource requirements. The optimized model was deployed as a video-upload-based web application prototype using Streamlit, allowing users to submit recorded sign-language clips for near-instantaneous prediction. This study concludes that the holistic skeletal approach is a promising and effective strategy for developing lightweight sign language translation systems on resource-constrained devices, although further validation with a larger vocabulary, repeated experimental trials, and real users is required prior to practical deployment.