THE TESTING OF FINE MOTOR SKILLS SYNERGIES BASED ON THE MEDIAPIPE HANDS NEURAL NETWORK AND THE FINGERFIT PRINCIPLE

The coordinated work of fingers is an indicator of human health and the key to professional skill. The research purpose was to create a method for testing of fine motor synergy based on computer vision. 

Methods and organization of the research. The proposed method is based on the use of the MediaPipe open source framework, namely the Mediapipe Hands neural network, which allows determining a person’s gesture based on the analysis of the video stream. Using a neural network and the author's method of assessing fine motor skills, we created a computer application FingerFit 4.0. That program allows tracking and analyzing changes in gestures and synergies in vivo. A 7-year-old girl, who has no health abnormalities or developmental delays, took part in the testing of the developed method. 

Results and discussion. The research showed that computer vision is able quickly and accurately detect the slightest changes in gesture. The test, which included 32 non-repeating gestures of one hand presented in random order, allowed identifying and evaluating 145 synergies. Based on the speed of gesture construction, synergies were ranking from the simplest to the most complex. Each derived synergy of the lower levels contributes to the formation of the synergy complexity of the highest level, although this phenomenon cannot be explained by simply adding up the synergetic load or increasing the number of fingers involved in the synergy.

Conclusion. The proposed method of fine motor synergy testing is easy to use and does not require additional expensive devices. A software application downloaded to a standard computer with a webcam allows a standardized and objective assessment of the state of fine motor skills.

• Patent 2717365 C1 Russian Federation. [A method for assessing fine motor skills of hands]: № 2018147383: A.A. Pomerancev, A.N. Starkin; Lipetsk State Pedagogical P. Semenov-Tyan-Shansky University. – № 2018147383.
• Subbotin, A. A., Voronova L. I. [The use of machine learning in gesture recognition tasks]. Telecommunications and information technology, 2022, Vol. 9, № 1, pp. 58-64.
• Hasanshin, I. YA. [Investigation of the kinematics of punches in karate based on an artificial neural network]. Science and sport: current trends, 2021, Vol. 9. № 1, pp. 36-42.
• Hasanshin, I. YA. [Application of machine vision technologies for recognition of single straight blows in boxing]. Science and sport: current trends, 2022, Vol. 10. № 2, pp. 43-48. – DOI 10.36028/2308-8826-2022-10-2-43-48.
• Hall, M. [Combinatorics]. Moscow, Mir. – 1970. – 423 p.
• Evaluation of Finger Force Control Ability in terms of Multi-finger Synergy. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, pp. 1-11. doi:10.1109/tnsre.2019.2932440
• Veluri, R.K., Sree, S.R., Vanathi, A., et al. Hand Gesture Mapping Using MediaPipe Algorithm. Proceedings of Third International Conference on Communication, Computing and Electronics Systems . Lecture Notes in Electrical Engineering, 2022, Vol. 844. doi: 10.1007/978-981-16-8862-1_39
• Indriani, Harris, M., Agoes, A. S. Applying Hand Gesture Recognition for User Guide Application Using MediaPipe. Proceedings of the 2nd International Seminar of Science and Applied Technology (ISSAT 2021), pp. 101-108 https://doi.org/10.2991/aer.k.211106.017
• Kim, K., Xu, D., Park, J. Effect of Kinetic Degrees of Freedom on Multi-Finger Synergies and Task Performance during Force Production and Release Tasks. Scientific Reports, 2018, № 8, 12758. https://doi.org/10.1038/s41598-018-31136-8
• Latash, M. L. One more time about motor (and non-motor) synergies. Experimental Brain Research, 2021. Oct., 239(10), pp. 2951-2967. doi:10.1007/s00221-021-06188-4
• Lim, K.Y., Joan, H., Tew, Y. Computer Performance Evaluation for Virtual Classroom with Artificial Intelligence Features. International Conference on Digital Transformation and Applications (ICDXA), 2021, pp. 85-94. doi: https://doi.org/10.56453/icdxa.2021.1008
• Madarshahian, S., Latash, M. L. Synergies at the level of motor units in single-finger and multi-finger tasks. Experimental Brain Research, 2021, 239(9), pp. 2905-2923. doi:10.1007/s00221-021-06180-y
• Cuadra, C., Bartsch, A., Tiemann, P., Reschechtko, S., Latash, M. L. Multi-finger synergies and the muscular apparatus of the hand. Experimental Brain Research, 2018, 236(5), pp. 1383-1393. doi:10.1007/s00221-018-5231-5
• Caputo, A., Giachetti, A., Soso, S. et al. SHREC 2021: Skeleton-based hand gesture recognition in the wild. Computers & Graphics, 2021, 99, pp. 201-211. doi:10.1016/j.cag.2021.07.007
• Top, E. Fine motor skills and attention level of individuals with mild intellectual disability getting education in inclusive classrooms and special education schools. International Journal of Developmental Disabilities, 2021, pp. 1-8. doi:10.1080/20473869.2021.1953940
• Lakkapragada, A., Kline, A., Mutlu, O.C. et al. The Classification of Abnormal Hand Movement to Aid in Autism Detection: Machine Learning Study. JMIR Biomedical Engineering, 2022, 7(1), e33771. doi: 10.2196/33771
• Güney, G., Jansen, T.S., Dill, S., Schulz, J.B. et al. Video-Based Hand Movement Analysis of Parkinson Patients before and after Medication Using High-Frame-Rate Videos and MediaPipe. Sensors, 2022, 22, 7992. https://doi.org/10.3390/s22207992