Deep Learning and Uniform LBP Histograms for Position Recognition of Elderly People with Privacy Preservation
Keywords:
Fall detection, Deep Learning, Classification, Deep Feed Forward Neural Network, Local Binary Pattern HistogramAbstract
For the elderly population, falls are a vital health problem especially in the current context of home care for COVID-19 patients. Given the saturation of health structures, patients are quarantined, in order to prevent the spread of the disease. Therefore, it is highly desirable to have a dedicated monitoring system to adequately improve their independent living and significantly reduce assistance costs. A fall event is considered as a specific and brutal change of pose. Thus, human poses should be first identified in order to detect abnormal events. Prompted by the great results achieved by the deep neural networks, we proposed a new architecture for image classification based on local binary pattern (LBP) histograms for feature extraction. These features were then saved, instead of saving the whole image in the series of identified poses. We aimed to preserve privacy, which is highly recommended in health informatics. The novelty of this study lies in the recognition of individuals’ positions in video images avoiding the convolution neural networks (CNNs) exorbitant computational cost and Minimizing the number of necessary inputs when learning a recognition model. The obtained numerical results of our approach application are very promising compared to the results of using other complex architectures like the deep CNNs.
References
[1] Noury, N.; Fleury, A.; Rumeau, P.; Bourke, A. K.; Laighin, G. O.; Rialle, V.; Lundy, J. E. (2007, August). Fall detection-principles and methods, In 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 1663-1666, 2007. https://doi.org/10.1109/IEMBS.2007.4352627
[2] Hyndman, D.; Ashburn, A.; Stack, E. (2002). Fall events among people with stroke living in the community: circumstances of falls and characteristics of fallers, Archives of physical medicine and rehabilitation, 83(2), 165-170, 2002. https://doi.org/10.1053/apmr.2002.28030
[3] Heinrich, S.; Rapp, K.; Rissmann, U.; Becker, C.; Kí¶nig, H. H. (2010). Cost of falls in old age: a systematic review, Osteoporosis international, 21(6), 891-902, 2010. https://doi.org/10.1007/s00198-009-1100-1
[4] Herath, S.; Harandi, M.; Porikli, F. (2017). Going deeper into action recognition: A survey, Image and Vision Computing, 60, 4-21, 2017. https://doi.org/10.1016/j.imavis.2017.01.010
[5] Poppe, R. (2010). A survey on vision-based human action recognition, Image Vis Comput, 28(6), 976-990, 2010. https://doi.org/10.1016/j.imavis.2009.11.014
[6] Majd, M.; Safabakhsh, R. (2019). A motion-aware ConvLSTM network for action recognitions, Applied Intelligence, 49(7), 2515-2521, 2019. https://doi.org/10.1007/s10489-018-1395-8
[7] Srivastava, N.; Mansimov, E.; Salakhudinov, R. (2015, June). Unsupervised learning of video representations using lstms, In International conference on machine learning, 843-852, 2015.
[8] Ordóñez, F. J.; Roggen, D. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition, Sensors, 16(1), 115, 2016. https://doi.org/10.3390/s16010115
[9] Delgado-Escaño, R.; Castro, F. M.; Cózar, J. R., Marín-Jiménez, M. J.; Guil, N.; Casilari, E. (2020). A cross-dataset deep learning-based classifier for people fall detection and identification, Computer methods and programs in biomedicine, 184, 105265, 2020. https://doi.org/10.1016/j.cmpb.2019.105265
[10] Lu, N.; Wu, Y.; Feng, L.; Song, J. (2018). Deep learning for fall detection: Three-dimensional CNN combined with LSTM on video kinematic data, IEEE journal of biomedical and health informatics, 23(1), 314-323, 2018. https://doi.org/10.1109/JBHI.2018.2808281
[11] Simonyan, K.; Zisserman, A. (2014, December). Two-stream convolutional networks for action recognition in videos, In Proceedings of the 27th International Conference on Neural Information Processing Systems, 1, 568-576, 2014.
[12] Feichtenhofer, C.; Pinz, A.; Zisserman, A. (2016). Convolutional two-stream network fusion for video action recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, 1933-1941, 2016. https://doi.org/10.1109/CVPR.2016.213
[13] Xingjian, S.H.I.; Chen, Z.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting, In Advances in neural information processing systems, 802-810, 2015.
[14] Li, Z.; Gavrilyuk, K.; Gavves, E.; Jain, M.; Snoek, C.G. (2018). Video lstm convolves, attends and flows for action recognition, Computer Vision and Image Understanding, 166, 41-50, 2018. https://doi.org/10.1016/j.cviu.2017.10.011
[15] Ercolano, G.; Rossi, S. (2021). Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation, Intelligent Service Robotics, 14(2), 175-185, 2021. https://doi.org/10.1007/s11370-021-00358-7
[16] Wu, J.M.T.; Li, Z.; Herencsar, N.; Vo, B.; Lin, J.C.W. (2021). A graph-based CNN-LSTM stock price prediction algorithm with leading indicators, Multimedia Systems, 1-20, 2021. https://doi.org/10.1007/s00530-021-00758-w
[17] Kourtzi, Z.; Kanwisher, N. (2000). Activation in human MT/MST by static images with implied motion, Journal of cognitive neuroscience, 12(1), 48-55, 2000. https://doi.org/10.1162/08989290051137594
[18] Justus, D.; Brennan, J.; Bonner, S.; McGough, A.S. (2018, December). Predicting the computational cost of deep learning models, In 2018 IEEE international conference on big data, 3873-3882, 2018 https://doi.org/10.1109/BigData.2018.8622396
[19] Neshatpour, K.; Homayoun, H.; Sasan, A. (2019). Icnn: The iterative convolutional neural network, ACM Transactions on Embedded Computing Systems, 18(6), 1-27, 2019. https://doi.org/10.1145/3355553
[20] He, K.; Sun, J. (2015). Convolutional neural networks at constrained time cost, In Proceedings of the IEEE conference on computer vision and pattern recognition, 5353-5360, 2015. https://doi.org/10.1109/CVPR.2015.7299173
[21] Singh, P.; Verma, V.K.; Rai, P.; Namboodiri, V.P. (2019). Hetconv: Heterogeneous kernel-based convolutions for deep cnns, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4835-4844, 2019. https://doi.org/10.1109/CVPR.2019.00497
[22] Asif, U.; Mashford, B.; Von Cavallar, S.; Yohanandan, S.; Roy, S.; Tang, J.; Harrer, S. (2020, April). Privacy preserving human fall detection using video data, In Machine Learning for Health Workshop, 39-51, 2020.
[23] Iazzi, A.; Rziza, M.; Thami, R.O.H. (2021). Fall Detection System-Based Posture-Recognition for Indoor Environments, Journal of Imaging, 7(3), 42, 2021. https://doi.org/10.3390/jimaging7030042
[24] Ricciuti, M.; Spinsante, S.; Gambi, E. (2018). Accurate fall detection in a top view privacy preserving configuration, Sensors, 18(6), 1754, 2018. https://doi.org/10.3390/s18061754
[25] Huang, D.; Shan, C.; Ardabilian, M.; Wang, Y.; Chen, L. (2011). Local binary patterns and its application to facial image analysis: a survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(6), 765-781, 2011. https://doi.org/10.1109/TSMCC.2011.2118750
[26] Dornaika, F.; Bosaghzadeh, A.; Salmane, H.; Ruichek, Y. (2014). A graph construction method using LBP self-representativeness for outdoor object categorization, Engineering Applications of Artificial Intelligence, 36, 294-302, 2014. https://doi.org/10.1016/j.engappai.2014.08.003
[27] Dornaika, F.; Bosaghzadeh, A.; Salmane. H.; Ruichek, Y. (2014). Graph-based semi-supervised learning with Local Binary Patterns for holistic object categorization, Expert Systems with Applications, 41(17), 7744-7753, 2014. https://doi.org/10.1016/j.eswa.2014.06.025
[28] Dornaika, F.; Bosaghzadeh, A. (2015). Adaptive graph construction using data selfrepresentativeness for pattern classification, Information Sciences, 325, 118-139, 2015. https://doi.org/10.1016/j.ins.2015.07.005
[29] Dornaika, F.; Moujahid, A.; El Merabet, Y.; Ruichek, Y. (2016). Building detection from orthophotos using a machine learning approach: An empirical study on image segmentation and descriptors, Expert Systems with Applications, 58, 130-142, 2016. https://doi.org/10.1016/j.eswa.2016.03.024
[30] Jebara, T.; Wang, J.; Chang, S.F. (2009). Graph construction and b-matching for semi-supervised learning, In Proceedings of the 26th annual international conference on machine learning, 441- 448, 2009. https://doi.org/10.1145/1553374.1553432
[31] Wright, J.; Yang, A.Y.; Ganesh, A.; Sastry, S.S.; Ma, Y. (2009). Robust face recognition via sparse representation, IEEE transactions on pattern analysis and machine intelligence, 31(2), 210-227, 2009. https://doi.org/10.1109/TPAMI.2008.79
[32] Adhikari, K.; Bouchachia, H.; Nait-Charif, H. (2017, May). Activity recognition for indoor fall detection using convolutional neural network, In 2017 Fifteenth IAPR International Conference on Machine Vision Applications, 81-84, 2017. https://doi.org/10.23919/MVA.2017.7986795
[33] Cook, D. (2016). Practical machine learning with H2O: powerful, scalable techniques for deep learning and AI, O'Reilly Media, Inc., 2016.
[34] Kever, I.; Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 15(1), 1929-1958, 2014.
[35] Brownlee, J. (2018). Better deep learning: train faster, reduce overfitting, and make better predictions, Machine Learning Mastery, 2018.
[36] Brownlee, J. (2016). Deep learning with Python: develop deep learning models on Theano and TensorFlow using Keras, Machine Learning Mastery, 2016.
[37] Bouhamed, H.; Ruichek, Y. (2018). Deep feedforward neural network learning using Local Binary Patterns histograms for outdoor object categorization, Advances In Modelling And Analyses B, 61(3), 158-162, 2018. https://doi.org/10.18280/ama_b.610309
[2] Hyndman, D.; Ashburn, A.; Stack, E. (2002). Fall events among people with stroke living in the community: circumstances of falls and characteristics of fallers, Archives of physical medicine and rehabilitation, 83(2), 165-170, 2002. https://doi.org/10.1053/apmr.2002.28030
[3] Heinrich, S.; Rapp, K.; Rissmann, U.; Becker, C.; Kí¶nig, H. H. (2010). Cost of falls in old age: a systematic review, Osteoporosis international, 21(6), 891-902, 2010. https://doi.org/10.1007/s00198-009-1100-1
[4] Herath, S.; Harandi, M.; Porikli, F. (2017). Going deeper into action recognition: A survey, Image and Vision Computing, 60, 4-21, 2017. https://doi.org/10.1016/j.imavis.2017.01.010
[5] Poppe, R. (2010). A survey on vision-based human action recognition, Image Vis Comput, 28(6), 976-990, 2010. https://doi.org/10.1016/j.imavis.2009.11.014
[6] Majd, M.; Safabakhsh, R. (2019). A motion-aware ConvLSTM network for action recognitions, Applied Intelligence, 49(7), 2515-2521, 2019. https://doi.org/10.1007/s10489-018-1395-8
[7] Srivastava, N.; Mansimov, E.; Salakhudinov, R. (2015, June). Unsupervised learning of video representations using lstms, In International conference on machine learning, 843-852, 2015.
[8] Ordóñez, F. J.; Roggen, D. (2016). Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition, Sensors, 16(1), 115, 2016. https://doi.org/10.3390/s16010115
[9] Delgado-Escaño, R.; Castro, F. M.; Cózar, J. R., Marín-Jiménez, M. J.; Guil, N.; Casilari, E. (2020). A cross-dataset deep learning-based classifier for people fall detection and identification, Computer methods and programs in biomedicine, 184, 105265, 2020. https://doi.org/10.1016/j.cmpb.2019.105265
[10] Lu, N.; Wu, Y.; Feng, L.; Song, J. (2018). Deep learning for fall detection: Three-dimensional CNN combined with LSTM on video kinematic data, IEEE journal of biomedical and health informatics, 23(1), 314-323, 2018. https://doi.org/10.1109/JBHI.2018.2808281
[11] Simonyan, K.; Zisserman, A. (2014, December). Two-stream convolutional networks for action recognition in videos, In Proceedings of the 27th International Conference on Neural Information Processing Systems, 1, 568-576, 2014.
[12] Feichtenhofer, C.; Pinz, A.; Zisserman, A. (2016). Convolutional two-stream network fusion for video action recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, 1933-1941, 2016. https://doi.org/10.1109/CVPR.2016.213
[13] Xingjian, S.H.I.; Chen, Z.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting, In Advances in neural information processing systems, 802-810, 2015.
[14] Li, Z.; Gavrilyuk, K.; Gavves, E.; Jain, M.; Snoek, C.G. (2018). Video lstm convolves, attends and flows for action recognition, Computer Vision and Image Understanding, 166, 41-50, 2018. https://doi.org/10.1016/j.cviu.2017.10.011
[15] Ercolano, G.; Rossi, S. (2021). Combining CNN and LSTM for activity of daily living recognition with a 3D matrix skeleton representation, Intelligent Service Robotics, 14(2), 175-185, 2021. https://doi.org/10.1007/s11370-021-00358-7
[16] Wu, J.M.T.; Li, Z.; Herencsar, N.; Vo, B.; Lin, J.C.W. (2021). A graph-based CNN-LSTM stock price prediction algorithm with leading indicators, Multimedia Systems, 1-20, 2021. https://doi.org/10.1007/s00530-021-00758-w
[17] Kourtzi, Z.; Kanwisher, N. (2000). Activation in human MT/MST by static images with implied motion, Journal of cognitive neuroscience, 12(1), 48-55, 2000. https://doi.org/10.1162/08989290051137594
[18] Justus, D.; Brennan, J.; Bonner, S.; McGough, A.S. (2018, December). Predicting the computational cost of deep learning models, In 2018 IEEE international conference on big data, 3873-3882, 2018 https://doi.org/10.1109/BigData.2018.8622396
[19] Neshatpour, K.; Homayoun, H.; Sasan, A. (2019). Icnn: The iterative convolutional neural network, ACM Transactions on Embedded Computing Systems, 18(6), 1-27, 2019. https://doi.org/10.1145/3355553
[20] He, K.; Sun, J. (2015). Convolutional neural networks at constrained time cost, In Proceedings of the IEEE conference on computer vision and pattern recognition, 5353-5360, 2015. https://doi.org/10.1109/CVPR.2015.7299173
[21] Singh, P.; Verma, V.K.; Rai, P.; Namboodiri, V.P. (2019). Hetconv: Heterogeneous kernel-based convolutions for deep cnns, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4835-4844, 2019. https://doi.org/10.1109/CVPR.2019.00497
[22] Asif, U.; Mashford, B.; Von Cavallar, S.; Yohanandan, S.; Roy, S.; Tang, J.; Harrer, S. (2020, April). Privacy preserving human fall detection using video data, In Machine Learning for Health Workshop, 39-51, 2020.
[23] Iazzi, A.; Rziza, M.; Thami, R.O.H. (2021). Fall Detection System-Based Posture-Recognition for Indoor Environments, Journal of Imaging, 7(3), 42, 2021. https://doi.org/10.3390/jimaging7030042
[24] Ricciuti, M.; Spinsante, S.; Gambi, E. (2018). Accurate fall detection in a top view privacy preserving configuration, Sensors, 18(6), 1754, 2018. https://doi.org/10.3390/s18061754
[25] Huang, D.; Shan, C.; Ardabilian, M.; Wang, Y.; Chen, L. (2011). Local binary patterns and its application to facial image analysis: a survey, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 41(6), 765-781, 2011. https://doi.org/10.1109/TSMCC.2011.2118750
[26] Dornaika, F.; Bosaghzadeh, A.; Salmane, H.; Ruichek, Y. (2014). A graph construction method using LBP self-representativeness for outdoor object categorization, Engineering Applications of Artificial Intelligence, 36, 294-302, 2014. https://doi.org/10.1016/j.engappai.2014.08.003
[27] Dornaika, F.; Bosaghzadeh, A.; Salmane. H.; Ruichek, Y. (2014). Graph-based semi-supervised learning with Local Binary Patterns for holistic object categorization, Expert Systems with Applications, 41(17), 7744-7753, 2014. https://doi.org/10.1016/j.eswa.2014.06.025
[28] Dornaika, F.; Bosaghzadeh, A. (2015). Adaptive graph construction using data selfrepresentativeness for pattern classification, Information Sciences, 325, 118-139, 2015. https://doi.org/10.1016/j.ins.2015.07.005
[29] Dornaika, F.; Moujahid, A.; El Merabet, Y.; Ruichek, Y. (2016). Building detection from orthophotos using a machine learning approach: An empirical study on image segmentation and descriptors, Expert Systems with Applications, 58, 130-142, 2016. https://doi.org/10.1016/j.eswa.2016.03.024
[30] Jebara, T.; Wang, J.; Chang, S.F. (2009). Graph construction and b-matching for semi-supervised learning, In Proceedings of the 26th annual international conference on machine learning, 441- 448, 2009. https://doi.org/10.1145/1553374.1553432
[31] Wright, J.; Yang, A.Y.; Ganesh, A.; Sastry, S.S.; Ma, Y. (2009). Robust face recognition via sparse representation, IEEE transactions on pattern analysis and machine intelligence, 31(2), 210-227, 2009. https://doi.org/10.1109/TPAMI.2008.79
[32] Adhikari, K.; Bouchachia, H.; Nait-Charif, H. (2017, May). Activity recognition for indoor fall detection using convolutional neural network, In 2017 Fifteenth IAPR International Conference on Machine Vision Applications, 81-84, 2017. https://doi.org/10.23919/MVA.2017.7986795
[33] Cook, D. (2016). Practical machine learning with H2O: powerful, scalable techniques for deep learning and AI, O'Reilly Media, Inc., 2016.
[34] Kever, I.; Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 15(1), 1929-1958, 2014.
[35] Brownlee, J. (2018). Better deep learning: train faster, reduce overfitting, and make better predictions, Machine Learning Mastery, 2018.
[36] Brownlee, J. (2016). Deep learning with Python: develop deep learning models on Theano and TensorFlow using Keras, Machine Learning Mastery, 2016.
[37] Bouhamed, H.; Ruichek, Y. (2018). Deep feedforward neural network learning using Local Binary Patterns histograms for outdoor object categorization, Advances In Modelling And Analyses B, 61(3), 158-162, 2018. https://doi.org/10.18280/ama_b.610309
Additional Files
Published
2021-09-16
Issue
Section
Articles
License
ONLINE OPEN ACCES: Acces to full text of each article and each issue are allowed for free in respect of Attribution-NonCommercial 4.0 International (CC BY-NC 4.0.
You are free to:
-Share: copy and redistribute the material in any medium or format;
-Adapt: remix, transform, and build upon the material.
The licensor cannot revoke these freedoms as long as you follow the license terms.
DISCLAIMER: The author(s) of each article appearing in International Journal of Computers Communications & Control is/are solely responsible for the content thereof; the publication of an article shall not constitute or be deemed to constitute any representation by the Editors or Agora University Press that the data presented therein are original, correct or sufficient to support the conclusions reached or that the experiment design or methodology is adequate.
