Sign Language Interpreter: A Mobile-Based Communication Application for Bridging the Communication Gap Between Deaf and Hearing Individuals
Downloads
Communication between deaf and hearing individuals remains a persistent challenge despite advances in mobile and machine-learning technologies. Existing sign language interpretation tools often suffer from limited vocabulary, dependence on specialized hardware, lack of integrated text-to-speech (TTS) functionality, environmental sensitivity, and the absence of offline capability.
This study aimed to design, develop, and evaluate a mobile-based Android application capable of (a) translating American Sign Language (ASL) gestures into text and synthesized speech in real time, (b) providing a built-in tutorial of the alphabet and commonly used words, and (c) operating offline on standard Android smartphones.
A descriptive-developmental research design was employed, complemented by an Agile software development life cycle (Define–Design–Develop–Demonstrate–Release). The application was implemented in Java using Android Studio, with hand-landmark detection performed by Google's MediaPipe framework. Quality assurance was conducted through automated tools (Sauce Labs, BrowserStack, WeTest, SonarQube) covering functionality, usability, performance, security, compatibility, reliability, and maintainability. Manual testing examined the influence of lighting, background, hand position, and hand size. User Acceptance Testing (UAT) was administered to 30 purposively sampled respondents (15 deaf, 15 hearing). Data were analyzed using percentages, weighted mean, and standard deviation.
Functionality testing yielded a 90.0% pass rate. Security testing revealed no high- or medium-risk vulnerabilities. Compatibility testing confirmed seamless operation on Android 12, 13, and 14 (rated Excellent) and acceptable performance on Android 11 (rated Good). Recognition accuracy was approximately 75% under controlled conditions, with optimal performance observed in natural light, against solid backgrounds, and with the hand placed frontally to the camera. UAT results across the eight ISO/IEC 25010 quality characteristics (Functional Suitability, Performance Efficiency, Compatibility, Usability, Reliability, Security, Maintainability, and Portability) ranged from 4.55 to 4.72 on a 4-point scale, all corresponding to a "Strongly Agree" verbal interpretation.
The developed mobile application addresses key gaps identified in prior literature by integrating real-time gesture-to-text translation, on-device text-to-speech synthesis, an embedded tutorial, and offline functionality on standard Android hardware. While the system performs reliably under controlled conditions, future work should focus on expanding the lexicon, accommodating dynamic gestures, improving robustness to lighting and background variability, and supporting cross-platform deployment.
Abalos, D. M., Dacsil, J. C. B., & Raquedan, A. A. L. (2021). SignTalk: Real-time mobile for the hearing-impaired individuals through a MediaPipe Holistic-based approach.
Abraham, A., & Rohini, V. (2018). Real-time conversion of sign language to speech and prediction of gestures using artificial neural network. Procedia Computer Science, 143, 587–594. https://doi.org/10.1016/j.procs.2018.10.435
Adeyanju, I. A., Bello, O. O., & Adegboye, M. A. (2021). Machine learning methods for sign language recognition: A critical review and analysis. Intelligent Systems with Applications, 12, 200056. https://doi.org/10.1016/j.iswa.2021.200056
Alam, S., Hussain, M., Munir, M. B., Shalahuddin, M., Ishrak, S., & Islam, M. N. (2021). A machine learning based sign language interpretation system for communication with deaf-mute people. Proceedings of the International Conference on Computing Advancements. https://doi.org/10.1145/3471391.3471422
Ardiansyah, A., Hitoyoshi, B., Halim, M., Hanafiah, N., & Wibisurya, A. (2021). Systematic literature review: American sign language translator. Procedia Computer Science, 179, 541–549. https://doi.org/10.1016/j.procs.2021.01.038
Azis, S. N. (2021). Handsign: Learn a sign language with your camera. DEV Community.
Bachchhav, P., & Potgantwar, A. (2019). Sign language interpreter using Kinect motion sensor using machine learning. International Journal of Innovative Technology and Exploring Engineering, 8(12).
Batte, B. (2017). A mobile-based system for translating sign language and facilitating communication between deaf and non-deaf people. Social Science Research Network. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4553096
Bisht, D., Kojage, M., Shukla, M., Patil, Y., & Bagade, P. (2022). Smart communication system using sign language interpretation. In 2022 31st Conference of Open Innovations Association (FRUCT). IEEE. https://doi.org/10.23919/FRUCT54823.2022.9770914
Brindha, R., Renukadevi, P., Vathana, D., & Jayanthi, D. (2022). Sign language interpreter. In 2022 International Conference on Inventive Computation Technologies (ICICT) (pp. 1024–1028). IEEE.
Chandragandhi, S., R, A. R., Ml, M. S., Akhil, S., & Pt, P. (2021). Real-time translation of sign language to speech and text. International Advanced Research Journal in Science, Engineering and Technology, 8(4). https://doi.org/10.17148/IARJSET.2021.8412
Chong, T. W., & Lee, B. (2018). American sign language recognition using Leap Motion controller with machine learning approach. Sensors, 18(10), 3554. https://doi.org/10.3390/s18103554
Halawani, S. M. (2008). Arabic sign language translation system on mobile devices. International Journal of Computer Science and Network Security, 8(1), 251–256.
Harkude, A., Namade, S., Patil, S., & Morey, A. (2020). Audio to sign language translation for deaf people. International Journal of Engineering and Innovative Technology, 9(10), 30.
Johnny, S., & Nirmala, S. J. (2021). Sign language translator using machine learning. SN Computer Science, 3(1). https://doi.org/10.1007/s42979-021-00896-y
Lee, C., & Gao, Z. (2020). Sign language recognition using two-stream convolutional neural networks with Wi-Fi signals. Applied Sciences, 10(24), 9005. https://doi.org/10.3390/app10249005
Mannan, A., Abbasi, A., Javed, A. R., Ahsan, A., Gadekallu, T. R., & Xin, Q. (2022). Hypertuned deep convolutional neural network for sign language recognition. Computational Intelligence and Neuroscience, 2022, 1–10. https://doi.org/10.1155/2022/1450822
Mohsin, S. (2024). Deep-learning approaches for static sign language recognition: A review. [Bibliographic details require verification].
Nandi, U., Shinde, A., & Itkarkar, R. R. (2017). Static and dynamic hand gesture recognition: A comparative study. International Journal of Engineering Research and Technology, 6(8), 23–27.
Nyaga, C., & Wario, R. (2018). Sign language gesture recognition through computer vision. In 2018 IST-Africa Week Conference (IST-Africa). IEEE.
Obi, Y., Claudio, K. S., Budiman, V. M., Achmad, S., & Kurniawan, A. (2023). Sign language recognition system for communicating to people with disabilities. Procedia Computer Science, 216, 13–20. https://doi.org/10.1016/j.procs.2022.12.106
Papastratis, I., Chatzikonstantinou, C., Konstantinidis, D., Dimitropoulos, K., & Daras, P. (2021). Artificial intelligence technologies for sign language. Sensors, 21(17), 5843. https://doi.org/10.3390/s21175843
Papastratis, I., Dimitropoulos, K., & Daras, P. (2021). Continuous sign language recognition through a context-aware generative adversarial network. Sensors, 21(7), 2437. https://doi.org/10.3390/s21072437
Papatsimouli, M., Sarigiannidis, P., & Fragulis, G. F. (2023). A survey of advancements in real-time sign language translators: Integration with IoT technology. Technologies, 11(4), 83. https://doi.org/10.3390/technologies11040083
Parton, B. S. (2005). Sign language recognition and translation: A multidisciplined approach from the field of artificial intelligence. The Journal of Deaf Studies and Deaf Education, 11(1), 94–101. https://doi.org/10.1093/deafed/enj003
Raboy, L., Canlas, J., Renejane, C., Mojica, C., & Bandiala, A. (2015). American sign language alphabet translator Android application: Hand shapes into text. Social Science Research Network. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3097464
Rajasekhar, N., Yadav, M., Vedantam, C., Pellakuru, K., & Navapete, C. (2023). Sign language recognition using machine learning algorithm. In 2023 International Conference on Sustainable Computing and Smart Systems (ICSCSS). IEEE. https://doi.org/10.1109/ICSCSS57650.2023.10169820
Rewari, H., Dixit, V., Batra, D., & Hema, N. (2018). Automated sign language interpreter. ResearchGate. https://www.researchgate.net/publication/328912139
RockCat Studio. (2020). AI Sign: Sign language [Mobile application].
Sanaullah, M., Ahmad, B., Kashif, M., Safdar, T., Hassan, M., Hasan, M. H., & Aziz, N. (2022). A real-time automatic translation of text to sign language. Computers, Materials & Continua, 70(2), 2471–2488.
Shukor, A. Z., Miskon, M. F., Jamaluddin, M. H., Bin Ali, F., Asyraf, M. F., & Bin Bahar, M. B. (2015). A new data glove approach for Malaysian sign language detection. Procedia Computer Science, 76, 60–67. https://doi.org/10.1016/j.procs.2015.12.276
Sood, K., Hernández, A., & Navdiya, B. (2023). American sign language interpreter: A bridge between the two worlds. In 2023 International Conference on Artificial Intelligence and Smart Communication (AISC). IEEE. https://doi.org/10.1109/AISC56616.2023.10085066
Vishwanath, S., & Yawer, S. S. (2019). Sign language interpreter using computer vision and LeNet-5 convolutional neural network architecture. International Journal of Innovative Science and Research Technology.
