3D Hand Gesture to Text and Speech Conversion Using Arduino and IMU Sensor
Keywords:
3D hand gesture recognition, IMU sensor, MPU6050, Arduino UNO, gesture-to-text conversion, gesture-to-speech conversion, wearable assistive device, Bluetooth HC-05, embedded system, real-time gesture processing, human-computer interactionAbstract
A hand gesture to speech and text conversion system in 3D that was designed to enhance communication with people with speech impairment. The system has been using a gesture glove, which is a wearable device, and it is connected to an IMU sensor (MPU6050) to record the actual hand orientation, motion, and finger gesture in real time. A sensor data is processed by an Arduino UNO which recognizes the predetermined gestures based on a threshold-based classification. The identified gestures appear on an LCD of size 16x 2 in form of text that is sent wirelessly via a Bluetooth (HC-05) module to a mobile phone to be read out. Buzzer will act as instant feedback when there is a successful gesture recognition and a battery supply is used to create portability. Serious outcomes of the experiments show that the system is an effective low-cost, user-friendly assistive communication aid to speech- and hearing-impaired users because its gesture recognition is performed in real-time with a low latency.
Downloads
References
P. Dhepekar and Y. G. Adhav, "Wireless robotic hand for remote operations using flex sensor," 2016 International Conference on Automatic Control and Dynamic Optimization Techniques (ICACDOT), Pune, 2016, pp. 114-118.
M. B. H. Flores, C. M. B. Siloy, C. Oppus and L. Agustin, "User-oriented finger-gesture glove controller with hand movement virtualization using flex sensors and a digital accelerometer," 2014 International Conference on Humanoid, Nanotechnology, Information Technology, Communication and Control, Environment and Management (HNICEM), Palawan, 2014, pp. 1-4.
L. Wang, T. Meydan, P. Williams and K. T. Wolfson, "A proposed optical-based sensor for assessment of hand movement," 2015 IEEE SENSORS, Busan, 2015, pp. 1-4. doi: 10.1109/ICSENS.2015.7370222
Manisha U. Kakde, Amit M. Rawate,” Hand Gesture Recognition System for Deaf and Dumb People Using PCA” International Journal of Engineering Science and Computing, July 2016
J. Liou and K. Fang, "Flex sensor for stroke patients identify the specific behavior with different bending situations," 2017 6th International Symposium on Next Generation Electronics (ISNE), Keelung, 2017, pp. 1-2. doi: 10.1109/ISNE.2017.7968716
Giovanni Saggio, Francesco Riillo, Laura Sbernini and Lucia Rita Quitadamo, “Resistive flex sensors: a survey”, Smart Materials and Structures, Vol. 25, number 1 doi: 10.1088/0964-1726/25/1/013001
C. E. A. Quiapo and K. N. M. Ramos, "Development of a sign language translator using simplified tilt, flex and
L. Sbernini, A. Pallotti and G. Saggio, "Evaluation of a Stretch Sensor for its inedited application in tracking hand finger movements," 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Benevento, 2016, pp. 1-6. doi: 10.1109/MeMeA.2016.7533809
B. Luan and M. Sun, "A simulation study on a single-unit wireless EEG Sensor," 2015 41st Annual Northeast Biomedical Engineering Conference (NEBEC), Troy, NY, 2015, pp. 1-2. doi: 10.1109/NEBEC.2015.7117176
Ji Jun, Yu MengSun, Zhou YuBin and Jin ZhangRui, "A Wireless EEG Sensors System for Computer Assisted Detection of Alpha Wave in Sleep," 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, 2005, pp. 5351-5353. doi: 10.1109/IEMBS.2005.1615690
T. J. Sullivan, S. R. Deiss and G. Cauwenberghs, "A Low-Noise, Non-Contact EEG/ECG Sensor," 2007 IEEE Biomedical Circuits and Systems Conference, Montreal, Que., 2007, pp. 154-157. doi:10.1109/BIOCAS.2007.4463332
Y. M. Chi and G. Cauwenberghs, "Wireless Noncontact EEG/ECG Electrodes for Body Sensor Networks," 2010 International Conference on Body Sensor Networks, Singapore, 2010, pp. 297-301. doi: 10.1109/BSN.2010.52
Fabien Lotte. A Tutorial on EEG Signal Processing Techniques for Mental State Recognition in Brain-Computer Interfaces. Eduardo Reck Miranda; Julien Castet. Guide to Brain-Computer Music Interfacing, Springer, 2014.
D. Vishal, H. M. Aishwarya, K. Nishkala, B. T. Royan and T. K. Ramesh, "Sign Language to Speech Conversion," 2017 IEEE International Conference on Computational Intelligence and Computing Research (ICCIC), Coimbatore, 2017, pp. 1-4. doi: 10.1109/ICCIC.2017.8523832
Anant S., Veni S., “Sensor-based hand gesture control system for safe driving”, Journal of Advanced Research in Dynamical and Control Systems, 10 (3), pp. 690-698
Anant S., Veni S., "Safe driving using vision-based hand gesture recognition system in non-uniform illumination conditions", Journal of ICT Research and Applications, 12 (2), pp. 154-167.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 International Journal of Scientific Research in Science and Technology
This work is licensed under a Creative Commons Attribution 4.0 International License.
https://creativecommons.org/licenses/by/4.0
