Hybrid-ResEff: A ResNet50-EfficientNetB0 Feature-Fusion Framework for Static Urdu Alphabet Recognition in Pakistan Sign Language
DOI:
https://doi.org/10.53560/PPASA(63-3)721Keywords:
Pakistan Sign Language, Urdu Alphabet Recognition, Deep Learning, Transfer Learning, ResNet50, EfficientNetB0, Feature FusionAbstract
Pakistan Sign Language (PSL) recognition remains challenging because publicly available datasets are limited and several Urdu alphabet gestures are visually similar. Existing PSL studies provide limited evaluation of whether fusing multiple pretrained backbones improves recognition across individual alphabet classes. This study proposes Hybrid-ResEff, a ResNet50-EfficientNetB0 feature-fusion framework for static PSL Urdu alphabet recognition. A publicly available PSL dataset containing 38 Urdu alphabet gesture classes and approximately 1,480 RGB images was used. ResNet50 and EfficientNetB0 were used as frozen ImageNet-pretrained feature extractors, and their feature vectors were concatenated before classification to combine potentially complementary visual representations. Model evaluation used stratified five-fold cross-validation and weighted as well as class-wise performance metrics. Hybrid-ResEff achieved the best performance among the five evaluated models, with accuracy of 0.9715 ± 0.0026 and weighted F1-score of 0.9659 ± 0.0025 (mean ± standard deviation across folds). Compared with ResNet50, the strongest standalone baseline, accuracy increased by 2.59 percentage points and weighted F1-score by 0.0275. Class-wise F1-scores ranged from 0.9250 to 0.9970. These findings support the potential of feature fusion on the evaluated dataset, while its small size limits conclusions about generalization to unseen signers and external datasets.
References
1. World Health Organization. Deafness and hearing loss. World Health Organization, Geneva, Switzerland (2026). https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss (accessed 10 June 2026).
2. United Nations. International Day of Sign Languages, 23 September. United Nations (2017). https://www.un.org/en/observances/sign-languages-day (accessed 10 June 2026)
3. W.C. Stokoe. Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. Journal of Deaf Studies and Deaf Education 10(1): 3-37 (2005). https://doi.org/10.1093/deafed/eni001
4. W. Sandler and D. Lillo-Martin (Eds.). Sign Language and Linguistic Universals. Cambridge University Press (2006). https://doi.org/10.1017/CBO9781139163910
5. D. Bragg, O. Koller, M. Bellard, L. Berke, P. Boudreault, A. Braffort, N. Caselli, M. Huenerfauth, H. Kacorri, T. Verhoef, C. Vogler, and M.R. Morris. Sign language recognition, generation, and translation: an interdisciplinary perspective. Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '19, October 28–30, 2019, Pittsburgh, PA, USA pp. 16-31 (2019). https://doi.org/10.1145/3308561.3353774
6. M.A. Ahmed, B.B. Zaidan, A.A. Zaidan, and M.M. Salih. A review on systems-based sensory gloves for sign language recognition: state of the art between 2007 and 2017. Sensors 18(7): 2208 (2018). https://doi.org/10.3390/s18072208
7. M.J. Cheok, Z. Omar, and M.H. Jaward. A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics 10: 131-153 (2019). https://doi.org/10.1007/s13042-017-0705-5
8. M. Al-Qurishi, T. Khalid, and R. Souissi. Deep learning for sign language recognition: current techniques, benchmarks, and open issues. IEEE Access 9: 126917-126951 (2021). https://doi.org/10.1109/ACCESS.2021.3110912
9. M. Madhiarasan and P.P. Roy. A comprehensive review of sign language recognition: different types, modalities, and datasets. arXiv preprint arXiv:2204.03328 (2022). https://doi.org/10.48550/arXiv.2204.03328
10. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA pp. 770-778 (2016). https://doi.org/10.1109/CVPR.2016.90
11. M. Tan and Q.V. Le. EfficientNet: rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, California pp. 6105-6114 (2019). https://proceedings.mlr.press/v97/tan19a.html (accessed 11 June 2026)
12. Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng. Deep Long-Tailed Learning: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(9): 10795-10816 (2023). https://doi.org/10.1109/TPAMI.2023.3268118
13. H. He and E.A. Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering 21(9): 1263-1284 (2009). https://doi.org/10.1109/TKDE.2008.239
14. J.M. Johnson and T.M. Khoshgoftaar. Survey on deep learning with class imbalance. Journal of Big Data 6: 27 (2019). https://doi.org/10.1186/s40537-019-0192-5
15. A. Imran, A. Razzaq, I.A. Baig, A. Hussain, S. Shahid, and T.U. Rehman. Dataset of Pakistan Sign Language and automatic recognition of hand configuration of Urdu alphabet through machine learning. Data in Brief 36: 107021 (2021). https://doi.org/10.1016/j.dib.2021.107021
16. H. Zahid, M. Rashid, S. Hussain, F. Azim, S.A. Syed, and A. Saad. Recognition of Urdu sign language: a systematic review of the machine learning classification. PeerJ Computer Science 8: e883 (2022). https://doi.org/10.7717/peerj-cs.883
17. F. Shah, M.S. Shah, W. Akram, A. Manzoor, R.O. Mahmoud, and D.S. Abdelminaam. Sign language recognition using multiple kernel learning: a case study of Pakistan Sign Language. IEEE Access 9: 67548-67558 (2021). https://doi.org/10.1109/ACCESS.2021.3077386
18. M.S. Mirza, S.M. Munaf, F. Azim, S. Ali, and S.J. Khan. Vision-based Pakistani sign language recognition using bag-of-words and support vector machines. Scientific Reports 12: 21325 (2022). https://doi.org/10.1038/s41598-022-15864-6
19. H.M. Hamza and A. Wali. Pakistan sign language recognition: leveraging deep learning models with limited dataset. Machine Vision and Applications 34: 71 (2023). https://doi.org/10.1007/s00138-023-01429-8
20. S. Arooj, S. Altaf, S. Ahmad, H. Mahmoud, and A.S.N. Mohamed. Enhancing sign language recognition using CNN and SIFT: a case study on Pakistan sign language. Journal of King Saud University – Computer and Information Sciences 36(2): 101934 (2024). https://doi.org/10.1016/j.jksuci.2024.101934
21. A.A. Mujeeb, A.H. Khan, S. Khalid, M.S. Mirza, and S.J. Khan. A neural-network based web application on real-time recognition of Pakistani sign language. Engineering Applications of Artificial Intelligence 135: 108761 (2024). https://doi.org/10.1016/j.engappai.2024.108761
22. K. Das, F. Abid, J. Rasheed, Kamlish, T. Asuroglu, S. Alsubai, and S. Soomro. Enhancing communication accessibility: UrSL-CNN approach to Urdu sign language translation for hearing-impaired individuals. Computer Modeling in Engineering & Sciences 141(1): 689-711 (2024). https://doi.org/10.32604/cmes.2024.051335
23. S. Naeem, H. Salam, and M.A. Uddin. Pakistani word-level sign language recognition based on deep spatiotemporal network. Proceedings of the AAAI Symposium Series 6(1): 119-126 (2025). https://doi.org/10.1609/aaaiss.v6i1.36042
24. J. Shin, A.S.M. Miah, Y. Akiba, K. Hirooka, N. Hassan, and Y.S. Hwang. Korean sign language alphabet recognition through the integration of handcrafted and deep learning-based two-stream feature extraction approach. IEEE Access 12: 68303-68318 (2024). https://doi.org/10.1109/ACCESS.2024.3399839
25. J.J. Bird, A. Ekárt, and D.R. Faria. British Sign Language recognition via late fusion of computer vision and Leap Motion with transfer learning to American Sign Language. Sensors 20(18): 5151 (2020). https://doi.org/10.3390/s20185151
26. M.A. Hamza, A. Subahi, N.A. Alghanmi, and N. Nemri. Deep fusion based transfer learning with bald eagle search algorithm for sign language recognition. Scientific Reports 15: 32752 (2025). https://doi.org/10.1038/s41598-025-10660-4
27. K. Zuiderveld. Contrast Limited Adaptive Histogram Equalization. In: Graphics Gems IV. Chapter VIII.5. P.S. Heckbert (Ed.). Academic Press pp. 474-485 (1994). https://doi.org/10.1016/B978-0-12-336156-1.50061-6
28. P. Kakumanu, S. Makrogiannis, and N. Bourbakis. A survey of skin-color modeling and detection methods. Pattern Recognition 40(3): 1106-1122 (2007). https://doi.org/10.1016/j.patcog.2006.06.010
29. J. Canny. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8(6): 679-698 (1986). https://doi.org/10.1109/TPAMI.1986.4767851
30. S. Bates, T. Hastie, and R. Tibshirani. Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association 119(546): 1434-1445 (2024). https://doi.org/10.1080/01621459.2023.2197686
31. S.J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10): 1345-1359 (2010). https://doi.org/10.1109/TKDE.2009.191
32. J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? Advances in Neural Information Processing Systems 27 (NIPS ’14), NIPS Foundation (2014). https://doi.org/10.48550/arXiv.1411.1792
33. K. Bayoudh. A survey of multimodal hybrid deep learning for computer vision: Architectures, applications, trends, and challenges. Information Fusion 105: 102217 (2024). https://doi.org/10.1016/j.inffus.2023.102217
34. D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations pp. 1-15 (2015). https://arxiv.org/abs/1412.6980
35. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15: 1929-1958 (2014). https://www.jmlr.org/papers/v15/srivastava14a.html
36. L. Prechelt. Early stopping — but when? In: Neural Networks: Tricks of the Trade. G.B. Orr, and K-R. Müller (Eds.). Springer, Berlin, Heidelberg pp. 55-69 (1998). https://doi.org/10.1007/3-540-49430-8_3
37. D.M.W. Powers. Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation. Journal of Machine Learning Technologies 2(1): 37-63 (2011). https://doi.org/10.48550/arXiv.2010.16061
38. M. Sokolova and G. Lapalme. A systematic analysis of performance measures for classification tasks. Information Processing & Management 45(4): 427-437 (2009). https://doi.org/10.1016/j.ipm.2009.03.002
39. M.K. Vidhyalakshmi, K. Jagadeesh, B.S. Dharmaraj, N.P. Sankar, and R.G. Kumar. Indian Sign Language Recognition using transfer learning with Efficient Net. Smart Science 12(2): 269-280 (2024). https://doi.org/10.1080/23080477.2024.2320956
40. M.A. Hamza, A. Subahi, N.A. Alghanmi, and N. Nemri. Deep fusion based transfer learning with bald eagle search algorithm for sign language recognition to assist individuals with hearing and speech impairments. Scientific Reports 15: 32752 (2025). https://doi.org/10.1038/s41598-025-10660-4
41. T. M. Shorten and T. M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data 6: 60 (2019). https://doi.org/10.1186/s40537-019-0197-0
42. J. Qin and M. Wang. Sign language recognition based on dual-channel star-attention convolutional neural network. Scientific Reports 15: 27685 (2025). https://doi.org/10.1038/s41598-025-13625-9
43. A. Boulesnane, L. Bellil, and M.G. Ghiri. A hybrid CNN-random forest model with landmark angles for real-time Arabic sign language recognition. Neural Computing and Applications 37: 2641-2662 (2025). https://doi.org/10.1007/s00521-024-10729-7
44. T.Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2): 318-327 (2020). https://doi.org/10.1109/TPAMI.2018.2858826
45. M. Hasan, B.K. Paul, N. Islam, and R. Mostafiz. Advancing real-time sign language detection for deaf and hearing-impaired communities: a customized YOLOv8 approach with tailored annotations in computer vision. BMC Artificial Intelligence 1: 11 (2025). https://doi.org/10.1186/s44398-025-00010-9
46. R. Yayla, H. Üçgün, and M. Abbas. Pose-Based Static Sign Language Recognition with Deep Learning for Turkish, Arabic, and American Sign Languages. Sensors 26(2): 524 (2026). https://doi.org/10.3390/s26020524
