|
1
|
Rudokaite J, Ong S, Onal Ertugrul I, Janssen MP, Huis in ‘t Veld E. Predicting vasovagal reactions to needles from video data using 2D-CNN with GRU and LSTM. PLoS One 2025; 20:e0314038. [PMID: 39854293 PMCID: PMC11760633 DOI: 10.1371/journal.pone.0314038] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Accepted: 11/04/2024] [Indexed: 01/26/2025] Open
Abstract
When undergoing or about to undergo a needle-related procedure, most people are not aware of the adverse emotional and physical reactions (so-called vasovagal reactions; VVR), that might occur. Thus, rather than relying on self-report measurements, we investigate whether we can predict VVR levels from the video sequence containing facial information measured during the blood donation. We filmed 287 blood donors throughout the blood donation procedure where we obtained 1945 videos for data analysis. We compared 5 different sequences of videos-45, 30, 20, 10 and 5 seconds to test the shortest video duration required to predict VVR levels. We used 2D-CNN with LSTM and GRU to predict continuous VVR scores and to classify discrete (low and high) VVR values obtained during the blood donation. The results showed that during the classification task, the highest achieved F1 score on high VVR class was 0.74 with a precision of 0.93, recall of 0.61, PR-AUC of 0.86 and an MCC score of 0.61 using a pre-trained ResNet152 model with LSTM on 25 frames and during the regression task the lowest root mean square error achieved was 2.56 using GRU on 50 frames. This study demonstrates that it is possible to predict vasovagal responses during a blood donation using facial features, which supports the further development of interventions to prevent VVR.
Collapse
Affiliation(s)
- Judita Rudokaite
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
- Donor Medicine Research, Sanquin Research, Amsterdam, The Netherlands
| | - Sharon Ong
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
| | - Itir Onal Ertugrul
- Department of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands
| | - Mart P. Janssen
- Donor Medicine Research, Sanquin Research, Amsterdam, The Netherlands
| | - Elisabeth Huis in ‘t Veld
- Department of Cognitive Science and Artificial Intelligence, Tilburg University, Tilburg, The Netherlands
- Donor Medicine Research, Sanquin Research, Amsterdam, The Netherlands
| |
Collapse
|
|
2
|
Lin J, Zhang L, Xia J, Zhang Y. Evaluating neonatal pain via fusing vision transformer and concept-cognitive computing. Sci Rep 2024; 14:26201. [PMID: 39482345 PMCID: PMC11528047 DOI: 10.1038/s41598-024-77521-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2024] [Accepted: 10/23/2024] [Indexed: 11/03/2024] Open
Abstract
In clinical nursing, neonatal pain assessment is a challenging task for preventing and controlling the impact of pain on neonatal development. To reduce the adverse effects of repetitive painful treatments during hospitalization on newborns, we propose a novel method (namely pain concept-cognitive computing model, PainC3M) for evaluating facial pain in newborns. In the fusion system, we first improve the attention mechanism of vision transformer by revising the node encoding way, considering the spatial structure, edge and centrality of nodes, and then use its corresponding encoder as a feature extractor to comprehensively extract image features. Second, we introduce a concept-cognitive computing model as a classifier to evaluate the level of pain. Finally, we evaluate our PainC3M on various open pain data sets and a real clinical pain data stream, and the experimental results demonstrate that our PainC3M is very effective for dynamic classification and superior to other comparative models. It also provides a good approach for pain assessment of individuals with aphasia (or dementia).
Collapse
Affiliation(s)
- Jing Lin
- School of Computer and Artificial Intelligence, Huaihua University, Huaihua, 418000, China.
| | - Liang Zhang
- School of Computer and Artificial Intelligence, Huaihua University, Huaihua, 418000, China
| | - Jianhua Xia
- School of Computer and Artificial Intelligence, Huaihua University, Huaihua, 418000, China
| | - Yuping Zhang
- Obstetrical Department of Huaihua Second People's Hospital, Huaihua, 418000, China
| |
Collapse
|
|
3
|
Meier TA, Refahi MS, Hearne G, Restifo DS, Munoz-Acuna R, Rosen GL, Woloszynek S. The Role and Applications of Artificial Intelligence in the Treatment of Chronic Pain. Curr Pain Headache Rep 2024; 28:769-784. [PMID: 38822995 DOI: 10.1007/s11916-024-01264-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/28/2024] [Indexed: 06/03/2024]
Abstract
PURPOSE OF REVIEW This review aims to explore the interface between artificial intelligence (AI) and chronic pain, seeking to identify areas of focus for enhancing current treatments and yielding novel therapies. RECENT FINDINGS In the United States, the prevalence of chronic pain is estimated to be upwards of 40%. Its impact extends to increased healthcare costs, reduced economic productivity, and strain on healthcare resources. Addressing this condition is particularly challenging due to its complexity and the significant variability in how patients respond to treatment. Current options often struggle to provide long-term relief, with their benefits rarely outweighing the risks, such as dependency or other side effects. Currently, AI has impacted four key areas of chronic pain treatment and research: (1) predicting outcomes based on clinical information; (2) extracting features from text, specifically clinical notes; (3) modeling 'omic data to identify meaningful patient subgroups with potential for personalized treatments and improved understanding of disease processes; and (4) disentangling complex neuronal signals responsible for pain, which current therapies attempt to modulate. As AI advances, leveraging state-of-the-art architectures will be essential for improving chronic pain treatment. Current efforts aim to extract meaningful representations from complex data, paving the way for personalized medicine. The identification of unique patient subgroups should reveal targets for tailored chronic pain treatments. Moreover, enhancing current treatment approaches is achievable by gaining a more profound understanding of patient physiology and responses. This can be realized by leveraging AI on the increasing volume of data linked to chronic pain.
Collapse
Affiliation(s)
| | - Mohammad S Refahi
- Ecological and Evolutionary Signal-Processing and Informatics (EESI) Laboratory, Department of Electrical and Computer Engineering, Drexel University, Philadelphia, PA, USA
| | - Gavin Hearne
- Ecological and Evolutionary Signal-Processing and Informatics (EESI) Laboratory, Department of Electrical and Computer Engineering, Drexel University, Philadelphia, PA, USA
| | | | - Ricardo Munoz-Acuna
- Anesthesia, Critical Care, and Pain Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA
| | - Gail L Rosen
- Ecological and Evolutionary Signal-Processing and Informatics (EESI) Laboratory, Department of Electrical and Computer Engineering, Drexel University, Philadelphia, PA, USA
| | - Stephen Woloszynek
- Anesthesia, Critical Care, and Pain Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA.
| |
Collapse
|
|
4
|
Eddy E, Campbell E, Bateman S, Scheme E. Understanding the influence of confounding factors in myoelectric control for discrete gesture recognition. J Neural Eng 2024; 21:036015. [PMID: 38722304 DOI: 10.1088/1741-2552/ad4915] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2024] [Accepted: 05/09/2024] [Indexed: 05/18/2024]
Abstract
Discrete myoelectric control-based gesture recognition has recently gained interest as a possible input modality for many emerging ubiquitous computing applications. Unlike the continuous control commonly employed in powered prostheses, discrete systems seek to recognize the dynamic sequences associated with gestures to generate event-based inputs. More akin to those used in general-purpose human-computer interaction, these could include, for example, a flick of the wrist to dismiss a phone call or a double tap of the index finger and thumb to silence an alarm. Moelectric control systems have been shown to achieve near-perfect classification accuracy, but in highly constrained offline settings. Real-world, online systems are subject to 'confounding factors' (i.e. factors that hinder the real-world robustness of myoelectric control that are not accounted for during typical offline analyses), which inevitably degrade system performance, limiting their practical use. Although these factors have been widely studied in continuous prosthesis control, there has been little exploration of their impacts on discrete myoelectric control systems for emerging applications and use cases. Correspondingly, this work examines, for the first time, three confounding factors and their effect on the robustness of discrete myoelectric control: (1)limb position variability, (2)cross-day use, and a newly identified confound faced by discrete systems (3)gesture elicitation speed. Results from four different discrete myoelectric control architectures: (1) Majority Vote LDA, (2) Dynamic Time Warping, (3) an LSTM network trained with Cross Entropy, and (4) an LSTM network trained with Contrastive Learning, show that classification accuracy is significantly degraded (p<0.05) as a result of each of these confounds. This work establishes that confounding factors are a critical barrier that must be addressed to enable the real-world adoption of discrete myoelectric control for robust and reliable gesture recognition.
Collapse
Affiliation(s)
- Ethan Eddy
- University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Evan Campbell
- University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Scott Bateman
- University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| | - Erik Scheme
- University of New Brunswick, Fredericton, NB E3B 5A3, Canada
| |
Collapse
|
|
5
|
Yue JM, Wang Q, Liu B, Zhou L. Postoperative accurate pain assessment of children and artificial intelligence: A medical hypothesis and planned study. World J Clin Cases 2024; 12:681-687. [PMID: 38322690 PMCID: PMC10841123 DOI: 10.12998/wjcc.v12.i4.681] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/27/2023] [Revised: 01/02/2023] [Accepted: 01/11/2024] [Indexed: 01/25/2024] Open
Abstract
Although the pediatric perioperative pain management has been improved in recent years, the valid and reliable pain assessment tool in perioperative period of children remains a challenging task. Pediatric perioperative pain management is intractable not only because children cannot express their emotions accurately and objectively due to their inability to describe physiological characteristics of feeling which are different from those of adults, but also because there is a lack of effective and specific assessment tool for children. In addition, exposure to repeated painful stimuli early in life is known to have short and long-term adverse sequelae. The short-term sequelae can induce a series of neurological, endocrine, cardiovascular system stress related to psychological trauma, while long-term sequelae may alter brain maturation process, which can lead to impair neurodevelopmental, behavioral, and cognitive function. Children's facial expressions largely reflect the degree of pain, which has led to the developing of a number of pain scoring tools that will help improve the quality of pain management in children if they are continually studied in depth. The artificial intelligence (AI) technology represented by machine learning has reached an unprecedented level in image processing of deep facial models through deep convolutional neural networks, which can effectively identify and systematically analyze various subtle features of children's facial expressions. Based on the construction of a large database of images of facial expressions in children with perioperative pain, this study proposes to develop and apply automatic facial pain expression recognition software using AI technology. The study aims to improve the postoperative pain management for pediatric population and the short-term and long-term quality of life for pediatric patients after operational event.
Collapse
Affiliation(s)
- Jian-Ming Yue
- Department of Anesthesiology, West China Hospital, Sichuan University, Chengdu 610041, Sichuan Province, China
| | - Qi Wang
- Department of Anesthesiology, West China Hospital, Sichuan University, Chengdu 610041, Sichuan Province, China
| | - Bin Liu
- Department of Anesthesiology, West China Hospital, Sichuan University, Chengdu 610041, Sichuan Province, China
| | - Leng Zhou
- Department of Anesthesiology, West China Hospital, Sichuan University, Chengdu 610041, Sichuan Province, China
| |
Collapse
|
|
6
|
Pinzon-Arenas JO, Kong Y, Chon KH, Posada-Quintero HF. Design and Evaluation of Deep Learning Models for Continuous Acute Pain Detection Based on Phasic Electrodermal Activity. IEEE J Biomed Health Inform 2023; 27:4250-4260. [PMID: 37399159 DOI: 10.1109/jbhi.2023.3291955] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/05/2023]
Abstract
The current method for assessing pain in clinical practice is subjective and relies on self-reported scales. An objective and accurate method of pain assessment is needed for physicians to prescribe the proper medication dosage, which could reduce addiction to opioids. Hence, many works have used electrodermal activity (EDA) as a suitable signal for detecting pain. Previous studies have used machine learning and deep learning to detect pain responses, but none have used a sequence-to-sequence deep learning approach to continuously detect acute pain from EDA signals, as well as accurate detection of pain onset. In this study, we evaluated deep learning models including 1-dimensional convolutional neural networks (1D-CNN), long short-term memory networks (LSTM), and three hybrid CNN-LSTM architectures for continuous pain detection using phasic EDA features. We used a database consisting of 36 healthy volunteers who underwent pain stimuli induced by a thermal grill. We extracted the phasic component, phasic drivers, and time-frequency spectrum of the phasic EDA (TFS-phEDA), which was found to be the most discerning physiomarker. The best model was a parallel hybrid architecture of a temporal convolutional neural network and a stacked bi-directional and uni-directional LSTM, which obtained a F1-score of 77.8% and was able to correctly detect pain in 15-second signals. The model was evaluated using 37 independent subjects from the BioVid Heat Pain Database and outperformed other approaches in recognizing higher pain levels compared to baseline with an accuracy of 91.5%. The results show the feasibility of continuous pain detection using deep learning and EDA.
Collapse
|
|
7
|
Li Y, He J, Fu C, Jiang K, Cao J, Wei B, Wang X, Luo J, Xu W, Zhu J. Children's Pain Identification Based on Skin Potential Signal. SENSORS (BASEL, SWITZERLAND) 2023; 23:6815. [PMID: 37571601 PMCID: PMC10422611 DOI: 10.3390/s23156815] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/04/2023] [Revised: 07/24/2023] [Accepted: 07/25/2023] [Indexed: 08/13/2023]
Abstract
Pain management is a crucial concern in medicine, particularly in the case of children who may struggle to effectively communicate their pain. Despite the longstanding reliance on various assessment scales by medical professionals, these tools have shown limitations and subjectivity. In this paper, we present a pain assessment scheme based on skin potential signals, aiming to convert subjective pain into objective indicators for pain identification using machine learning methods. We have designed and implemented a portable non-invasive measurement device to measure skin potential signals and conducted experiments involving 623 subjects. From the experimental data, we selected 358 valid records, which were then divided into 218 silent samples and 262 pain samples. A total of 38 features were extracted from each sample, with seven features displaying superior performance in pain identification. Employing three classification algorithms, we found that the random forest algorithm achieved the highest accuracy, reaching 70.63%. While this identification rate shows promise for clinical applications, it is important to note that our results differ from state-of-the-art research, which achieved a recognition rate of 81.5%. This discrepancy arises from the fact that our pain stimuli were induced by clinical operations, making it challenging to precisely control the stimulus intensity when compared to electrical or thermal stimuli. Despite this limitation, our pain assessment scheme demonstrates significant potential in providing objective pain identification in clinical settings. Further research and refinement of the proposed approach may lead to even more accurate and reliable pain management techniques in the future.
Collapse
Affiliation(s)
- Yubo Li
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
- International Joint Innovation Center, Zhejiang University, Haining 314400, China
| | - Jiadong He
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
| | - Cangcang Fu
- Children’s Hospital, Zhejiang University School of Medicine, Hangzhou 310052, China; (C.F.); (W.X.)
| | - Ke Jiang
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
| | - Junjie Cao
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
| | - Bing Wei
- Polytechnic Institute of Zhejiang University, Hangzhou 310015, China;
| | - Xiaozhi Wang
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
- International Joint Innovation Center, Zhejiang University, Haining 314400, China
| | - Jikui Luo
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
- International Joint Innovation Center, Zhejiang University, Haining 314400, China
| | - Weize Xu
- Children’s Hospital, Zhejiang University School of Medicine, Hangzhou 310052, China; (C.F.); (W.X.)
- National Clinical Research Center for Child Health, Hangzhou 310052, China
| | - Jihua Zhu
- College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310027, China; (J.H.); (K.J.); (J.C.); (X.W.); (J.L.)
- Children’s Hospital, Zhejiang University School of Medicine, Hangzhou 310052, China; (C.F.); (W.X.)
| |
Collapse
|
|
8
|
Cascella M, Schiavo D, Cuomo A, Ottaiano A, Perri F, Patrone R, Migliarelli S, Bignami EG, Vittori A, Cutugno F. Artificial Intelligence for Automatic Pain Assessment: Research Methods and Perspectives. Pain Res Manag 2023; 2023:6018736. [PMID: 37416623 PMCID: PMC10322534 DOI: 10.1155/2023/6018736] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Revised: 02/03/2023] [Accepted: 04/20/2023] [Indexed: 07/08/2023]
Abstract
Although proper pain evaluation is mandatory for establishing the appropriate therapy, self-reported pain level assessment has several limitations. Data-driven artificial intelligence (AI) methods can be employed for research on automatic pain assessment (APA). The goal is the development of objective, standardized, and generalizable instruments useful for pain assessment in different clinical contexts. The purpose of this article is to discuss the state of the art of research and perspectives on APA applications in both research and clinical scenarios. Principles of AI functioning will be addressed. For narrative purposes, AI-based methods are grouped into behavioral-based approaches and neurophysiology-based pain detection methods. Since pain is generally accompanied by spontaneous facial behaviors, several approaches for APA are based on image classification and feature extraction. Language features through natural language strategies, body postures, and respiratory-derived elements are other investigated behavioral-based approaches. Neurophysiology-based pain detection is obtained through electroencephalography, electromyography, electrodermal activity, and other biosignals. Recent approaches involve multimode strategies by combining behaviors with neurophysiological findings. Concerning methods, early studies were conducted by machine learning algorithms such as support vector machine, decision tree, and random forest classifiers. More recently, artificial neural networks such as convolutional and recurrent neural network algorithms are implemented, even in combination. Collaboration programs involving clinicians and computer scientists must be aimed at structuring and processing robust datasets that can be used in various settings, from acute to different chronic pain conditions. Finally, it is crucial to apply the concepts of explainability and ethics when examining AI applications for pain research and management.
Collapse
Affiliation(s)
- Marco Cascella
- Division of Anesthesia and Pain Medicine, Istituto Nazionale Tumori IRCCS Fondazione G. Pascale, Naples 80131, Italy
| | - Daniela Schiavo
- Division of Anesthesia and Pain Medicine, Istituto Nazionale Tumori IRCCS Fondazione G. Pascale, Naples 80131, Italy
| | - Arturo Cuomo
- Division of Anesthesia and Pain Medicine, Istituto Nazionale Tumori IRCCS Fondazione G. Pascale, Naples 80131, Italy
| | - Alessandro Ottaiano
- SSD-Innovative Therapies for Abdominal Metastases, Istituto Nazionale Tumori di Napoli IRCCS “G. Pascale”, Via M. Semmola, Naples 80131, Italy
| | - Francesco Perri
- Head and Neck Oncology Unit, Istituto Nazionale Tumori IRCCS-Fondazione “G. Pascale”, Naples 80131, Italy
| | - Renato Patrone
- Dieti Department, University of Naples, Naples, Italy
- Division of Hepatobiliary Surgical Oncology, Istituto Nazionale Tumori IRCCS, Fondazione Pascale-IRCCS di Napoli, Naples, Italy
| | - Sara Migliarelli
- Department of Pharmacology, Faculty of Medicine and Psychology, University Sapienza of Rome, Rome, Italy
| | - Elena Giovanna Bignami
- Anesthesiology, Critical Care and Pain Medicine Division, Department of Medicine and Surgery, University of Parma, Parma, Italy
| | - Alessandro Vittori
- Department of Anesthesia and Critical Care, ARCO ROMA, Ospedale Pediatrico Bambino Gesù IRCCS, Rome 00165, Italy
| | - Francesco Cutugno
- Department of Electrical Engineering and Information Technologies, University of Naples “Federico II”, Naples 80100, Italy
| |
Collapse
|
|
9
|
Davodabadi A, Daneshian B, Saati S, Razavyan S. Mathematical model and artificial intelligence for diagnosis of Alzheimer's disease. EUROPEAN PHYSICAL JOURNAL PLUS 2023; 138:474. [PMID: 37274456 PMCID: PMC10226030 DOI: 10.1140/epjp/s13360-023-04128-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 05/22/2023] [Indexed: 06/06/2023]
Abstract
Degeneration of the neurological system linked to cognitive deficits, daily living exercise clutters, and behavioral disturbing impacts may define Alzheimer's disease. Alzheimer's disease research conducted later in life focuses on describing ways for early detection of dementia, a kind of mental disorder. To tailor our care to each patient, we utilized visual cues to determine how they were feeling. We did this by outlining two approaches to diagnosing a person's mental health. Support vector machine is the first technique. Image characteristics are extracted using a fractal model for classification in this method. With this technique, the histogram of a picture is modeled after a Gaussian distribution. Classification was performed with several support vector machines kernels, and the outcomes were compared. Step two proposes using a deep convolutional neural network architecture to identify Alzheimer's-related mental disorders. According to the findings, the support vector machines approach accurately recognized over 93% of the photos tested. The deep convolutional neural network approach was one hundred percent accurate during model training, whereas the support vector machines approach achieved just 93 percent accuracy. In contrast to support vector machines accuracy of 89.3%, the deep convolutional neural network model test findings were accurate 98.8% of the time. Based on the findings reported here, the proposed deep convolutional neural network architecture may be used for diagnostic purposes involving the patient's mental state.
Collapse
Affiliation(s)
- Afsaneh Davodabadi
- Department of Mathematics, Central Tehran Branch, Islamic Azad University, Tehran, Iran
| | - Behrooz Daneshian
- Department of Mathematics, Central Tehran Branch, Islamic Azad University, Tehran, Iran
| | - Saber Saati
- Department of Mathematics, North Tehran Branch, Islamic Azad University, Tehran, Iran
| | - Shabnam Razavyan
- Department of Mathematics, South Tehran Branch, Islamic Azad University, Tehran, Iran
| |
Collapse
|
|
10
|
De Sario GD, Haider CR, Maita KC, Torres-Guzman RA, Emam OS, Avila FR, Garcia JP, Borna S, McLeod CJ, Bruce CJ, Carter RE, Forte AJ. Using AI to Detect Pain through Facial Expressions: A Review. Bioengineering (Basel) 2023; 10:548. [PMID: 37237618 PMCID: PMC10215219 DOI: 10.3390/bioengineering10050548] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2023] [Revised: 04/24/2023] [Accepted: 04/27/2023] [Indexed: 05/28/2023] Open
Abstract
Pain assessment is a complex task largely dependent on the patient's self-report. Artificial intelligence (AI) has emerged as a promising tool for automating and objectifying pain assessment through the identification of pain-related facial expressions. However, the capabilities and potential of AI in clinical settings are still largely unknown to many medical professionals. In this literature review, we present a conceptual understanding of the application of AI to detect pain through facial expressions. We provide an overview of the current state of the art as well as the technical foundations of AI/ML techniques used in pain detection. We highlight the ethical challenges and the limitations associated with the use of AI in pain detection, such as the scarcity of databases, confounding factors, and medical conditions that affect the shape and mobility of the face. The review also highlights the potential impact of AI on pain assessment in clinical practice and lays the groundwork for further study in this area.
Collapse
Affiliation(s)
| | - Clifton R. Haider
- Department of Physiology and Biomedical Engineering, Mayo Clinic, Rochester, MN 55902, USA
| | - Karla C. Maita
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | | | - Omar S. Emam
- Division of AI in Health Sciences, University of Louisville, Louisville, KY 40292, USA
| | | | - John P. Garcia
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Sahar Borna
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | | | - Charles J. Bruce
- Department of Cardiovascular Medicine, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Rickey E. Carter
- Department of Health Sciences Research, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Antonio J. Forte
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| |
Collapse
|
|
11
|
Liao J, Lin Y, Ma T, He S, Liu X, He G. Facial Expression Recognition Methods in the Wild Based on Fusion Feature of Attention Mechanism and LBP. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23094204. [PMID: 37177408 PMCID: PMC10180539 DOI: 10.3390/s23094204] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/26/2022] [Revised: 03/16/2023] [Accepted: 04/12/2023] [Indexed: 05/15/2023]
Abstract
Facial expression methods play a vital role in human-computer interaction and other fields, but there are factors such as occlusion, illumination, and pose changes in wild facial recognition, as well as category imbalances between different datasets, that result in large variations in recognition rates and low accuracy rates for different categories of facial expression datasets. This study introduces RCL-Net, a method of recognizing wild facial expressions that is based on an attention mechanism and LBP feature fusion. The structure consists of two main branches, namely the ResNet-CBAM residual attention branch and the local binary feature (LBP) extraction branch (RCL-Net). First, by merging the residual network and hybrid attention mechanism, the residual attention network is presented to emphasize the local detail feature information of facial expressions; the significant characteristics of facial expressions are retrieved from both channel and spatial dimensions to build the residual attention classification model. Second, we present a locally improved residual network attention model. LBP features are introduced into the facial expression feature extraction stage in order to extract texture information on expression photographs in order to emphasize facial feature information and enhance the recognition accuracy of the model. Lastly, experimental validation is performed using the FER2013, FERPLUS, CK+, and RAF-DB datasets, and the experimental results demonstrate that the proposed method has superior generalization capability and robustness in the laboratory-controlled environment and field environment compared to the most recent experimental methods.
Collapse
Affiliation(s)
- Jun Liao
- Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
- College of Mechanical Engineering, Chongqing University of Technology, Chongqing 400054, China
- Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
| | - Yuanchang Lin
- Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
- Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
| | - Tengyun Ma
- Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
- Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
| | - Songxiying He
- Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
| | - Xiaofang Liu
- Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
- Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
| | - Guotian He
- Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
- Chongqing Key Laboratory of Artificial Intelligence and Service Robot Control Technology, Chongqing Institute of Green Intelligent Technology, Chinese Academy of Sciences, Chongqing 400714, China
| |
Collapse
|
|
12
|
Borna S, Haider CR, Maita KC, Torres RA, Avila FR, Garcia JP, De Sario Velasquez GD, McLeod CJ, Bruce CJ, Carter RE, Forte AJ. A Review of Voice-Based Pain Detection in Adults Using Artificial Intelligence. Bioengineering (Basel) 2023; 10:bioengineering10040500. [PMID: 37106687 PMCID: PMC10135816 DOI: 10.3390/bioengineering10040500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2023] [Revised: 04/13/2023] [Accepted: 04/19/2023] [Indexed: 04/29/2023] Open
Abstract
Pain is a complex and subjective experience, and traditional methods of pain assessment can be limited by factors such as self-report bias and observer variability. Voice is frequently used to evaluate pain, occasionally in conjunction with other behaviors such as facial gestures. Compared to facial emotions, there is less available evidence linking pain with voice. This literature review synthesizes the current state of research on the use of voice recognition and voice analysis for pain detection in adults, with a specific focus on the role of artificial intelligence (AI) and machine learning (ML) techniques. We describe the previous works on pain recognition using voice and highlight the different approaches to voice as a tool for pain detection, such as a human effect or biosignal. Overall, studies have shown that AI-based voice analysis can be an effective tool for pain detection in adult patients with various types of pain, including chronic and acute pain. We highlight the high accuracy of the ML-based approaches used in studies and their limitations in terms of generalizability due to factors such as the nature of the pain and patient population characteristics. However, there are still potential challenges, such as the need for large datasets and the risk of bias in training models, which warrant further research.
Collapse
Affiliation(s)
- Sahar Borna
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Clifton R Haider
- Department of Physiology and Biomedical Engineering, Mayo Clinic, Rochester, MN 55902, USA
| | - Karla C Maita
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Ricardo A Torres
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Francisco R Avila
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | - John P Garcia
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| | | | | | - Charles J Bruce
- Department of Cardiovascular Medicine, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Rickey E Carter
- Department of Health Sciences Research, Mayo Clinic, Jacksonville, FL 32224, USA
| | - Antonio J Forte
- Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA
| |
Collapse
|
|
13
|
Image-Based Pain Intensity Estimation Using Parallel CNNs with Regional Attention. BIOENGINEERING (BASEL, SWITZERLAND) 2022; 9:bioengineering9120804. [PMID: 36551010 PMCID: PMC9774603 DOI: 10.3390/bioengineering9120804] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 11/28/2022] [Accepted: 12/09/2022] [Indexed: 12/23/2022]
Abstract
Automatic pain estimation plays an important role in the field of medicine and health. In the previous studies, most of the entire image frame was directly imported into the model. This operation can allow background differences to negatively affect the experimental results. To tackle this issue, we propose the parallel CNNs framework with regional attention for automatic pain intensity estimation at the frame level. This modified convolution neural network structure combines BlurPool methods to enhance translation invariance in network learning. The improved networks can focus on learning core regions while supplementing global information, thereby obtaining parallel feature information. The core regions are mainly based on the tradeoff between the weights of the channel attention modules and the spatial attention modules. Meanwhile, the background information of the non-core regions is shielded by the DropBlock algorithm. These steps enable the model to learn facial pain features adaptively, not limited to a single image pattern. The experimental result of our proposed model outperforms many state-of-the-art methods on the RMSE and PCC metrics when evaluated on the diverse pain levels of over 12,000 images provided by the publicly available UNBC dataset. The model accuracy rate has reached 95.11%. The experimental results show that the proposed method is highly efficient at extracting the facial features of pain and predicts pain levels with high accuracy.
Collapse
|
|
14
|
Xiang X, Wang F, Tan Y, Yuille AL. Imbalanced regression for intensity series of pain expression from videos by regularizing spatio-temporal face nets. Pattern Recognit Lett 2022. [DOI: 10.1016/j.patrec.2022.09.022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
|
15
|
Xu Z, Tang B, Zhang X, Leong JF, Pan J, Hooda S, Zamburg E, Thean AVY. Reconfigurable nonlinear photonic activation function for photonic neural network based on non-volatile opto-resistive RAM switch. LIGHT, SCIENCE & APPLICATIONS 2022; 11:288. [PMID: 36202804 PMCID: PMC9537414 DOI: 10.1038/s41377-022-00976-5] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Revised: 08/15/2022] [Accepted: 08/30/2022] [Indexed: 06/16/2023]
Abstract
Photonic neural network has been sought as an alternative solution to surpass the efficiency and speed bottlenecks of electronic neural network. Despite that the integrated Mach-Zehnder Interferometer (MZI) mesh can perform vector-matrix multiplication in photonic neural network, a programmable in-situ nonlinear activation function has not been proposed to date, suppressing further advancement of photonic neural network. Here, we demonstrate an efficient in-situ nonlinear accelerator comprising a unique solution-processed two-dimensional (2D) MoS2 Opto-Resistive RAM Switch (ORS), which exhibits tunable nonlinear resistance switching that allow us to introduce nonlinearity to the photonic neuron which overcomes the linear voltage-power relationship of typical photonic components. Our reconfigurable scheme enables implementation of a wide variety of nonlinear responses. Furthermore, we confirm its feasibility and capability for MNIST handwritten digit recognition, achieving a high accuracy of 91.6%. Our accelerator constitutes a major step towards the realization of in-situ photonic neural network and pave the way for the integration of photonic integrated circuits (PIC).
Collapse
Affiliation(s)
- Zefeng Xu
- Integrative Sciences and Engineering Programme, NUS Graduate School, National University of Singapore, Singapore, Singapore.
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore.
| | - Baoshan Tang
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore
| | - Xiangyu Zhang
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore
| | - Jin Feng Leong
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore
| | - Jieming Pan
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore
| | - Sonu Hooda
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore
| | - Evgeny Zamburg
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore
| | - Aaron Voon-Yew Thean
- Integrative Sciences and Engineering Programme, NUS Graduate School, National University of Singapore, Singapore, Singapore.
- Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore, 117583, Singapore.
| |
Collapse
|
|
16
|
Szczapa B, Daoudi M, Berretti S, Pala P, Del Bimbo A, Hammal Z. Automatic Estimation of Self-Reported Pain by Trajectory Analysis in the Manifold of Fixed Rank Positive Semi-Definite Matrices. IEEE TRANSACTIONS ON AFFECTIVE COMPUTING 2022; 13:1813-1826. [PMID: 36452255 PMCID: PMC9708064 DOI: 10.1109/taffc.2022.3207001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
We propose an automatic method to estimate self-reported pain based on facial landmarks extracted from videos. For each video sequence, we decompose the face into four different regions and the pain intensity is measured by modeling the dynamics of facial movement using the landmarks of these regions. A formulation based on Gram matrices is used for representing the trajectory of landmarks on the Riemannian manifold of symmetric positive semi-definite matrices of fixed rank. A curve fitting algorithm is used to smooth the trajectories and temporal alignment is performed to compute the similarity between the trajectories on the manifold. A Support Vector Regression classifier is then trained to encode extracted trajectories into pain intensity levels consistent with self-reported pain intensity measurement. Finally, a late fusion of the estimation for each region is performed to obtain the final predicted pain level. The proposed approach is evaluated on two publicly available datasets, the UNBCMcMaster Shoulder Pain Archive and the Biovid Heat Pain dataset. We compared our method to the state-of-the-art on both datasets using different testing protocols, showing the competitiveness of the proposed approach.
Collapse
Affiliation(s)
- Benjamin Szczapa
- Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France
| | - Mohamed Daoudi
- IMT Nord Europe, Institut Mines-Télécom, Univ. Lille, Centre for Digital Systems, F-59000 Lille, France, and Univ. Lille, CNRS, Centrale Lille, Institut Mines-Télécom, UMR 9189 CRIStAL, F-59000 Lille, France
| | - Stefano Berretti
- Department of Information Engineering, University of Florence, Italy
| | - Pietro Pala
- Department of Information Engineering, University of Florence, Italy
| | - Alberto Del Bimbo
- Department of Information Engineering, University of Florence, Italy
| | - Zakia Hammal
- Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA
| |
Collapse
|
|
17
|
Abdulghafor R, Turaev S, Ali MAH. Body Language Analysis in Healthcare: An Overview. Healthcare (Basel) 2022; 10:healthcare10071251. [PMID: 35885777 PMCID: PMC9325107 DOI: 10.3390/healthcare10071251] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2022] [Revised: 06/22/2022] [Accepted: 06/23/2022] [Indexed: 11/16/2022] Open
Abstract
Given the current COVID-19 pandemic, medical research today focuses on epidemic diseases. Innovative technology is incorporated in most medical applications, emphasizing the automatic recognition of physical and emotional states. Most research is concerned with the automatic identification of symptoms displayed by patients through analyzing their body language. The development of technologies for recognizing and interpreting arm and leg gestures, facial features, and body postures is still in its early stage. More extensive research is needed using artificial intelligence (AI) techniques in disease detection. This paper presents a comprehensive survey of the research performed on body language processing. Upon defining and explaining the different types of body language, we justify the use of automatic recognition and its application in healthcare. We briefly describe the automatic recognition framework using AI to recognize various body language elements and discuss automatic gesture recognition approaches that help better identify the external symptoms of epidemic and pandemic diseases. From this study, we found that since there are studies that have proven that the body has a language called body language, it has proven that language can be analyzed and understood by machine learning (ML). Since diseases also show clear and different symptoms in the body, the body language here will be affected and have special features related to a particular disease. From this examination, we discovered that it is possible to specialize the features and language changes of each disease in the body. Hence, ML can understand and detect diseases such as pandemic and epidemic diseases and others.
Collapse
Affiliation(s)
- Rawad Abdulghafor
- Department of Computer Science, Faculty of Information and Communication Technology, International Islamic University Malaysia, Kuala Lumpur 53100, Malaysia
- Correspondence: (R.A.); (S.T.); (M.A.H.A.)
| | - Sherzod Turaev
- Department of Computer Science and Software Engineering, College of Information Technology, United Arab Emirates University, Al-Ain, Abu Dhabi P.O. Box 15556, United Arab Emirates
- Correspondence: (R.A.); (S.T.); (M.A.H.A.)
| | - Mohammed A. H. Ali
- Department of Mechanical Engineering, Faculty of Engineering, University of Malaya, Kuala Lumpur 50603, Malaysia
- Correspondence: (R.A.); (S.T.); (M.A.H.A.)
| |
Collapse
|
|
18
|
Abstract
Pain assessment is used to improve patients’ treatment outcomes. Human observers may be influenced by personal factors, such as inexperience and medical organizations are facing a shortage of experts. In this study, we developed a facial expressions-based automatic pain assessment system (FEAPAS) to notify medical staff when a patient suffers pain by activating an alarm and recording the incident and pain level with the date and time. The model consists of two identical concurrent subsystems, each of which takes one of the two inputs of the model, i.e., “full face” and “the upper half of the same face”. The subsystems extract the relevant input features via two pre-trained convolutional neural networks (CNNs), using either VGG16, InceptionV3, ResNet50, or ResNeXt50, while freezing all convolutional blocks and replacing the classifier layer with a shallow CNN. The concatenated outputs in this stage is then sent to the model’s classifier. This approach mimics the human observer method and gives more importance to the upper part of the face, which is similar to the Prkachin and Soloman pain intensity (PSPI). Additionally, we further optimized our models by applying four optimizers (SGD/ADAM/RMSprop/RAdam) to each model and testing them on the UNBC-McMaster shoulder pain expression archive dataset to find the optimal combination, InceptionV3-SGD. The optimal model showed an accuracy of 99.10% on 10-fold cross-validation, thus outperforming the state-of-the-art model on the UNBC-McMaster database. It also scored 90.56% on unseen subject data. To speed up the system response time and reduce unnecessary alarms associated with temporary facial expressions, a select but effective subset of frames was inspected and classified. Two frame-selection criteria were reported. Classifying only two frames at the middle of 30-frame sequence was optimal, with an average reaction time of at most 6.49 s and the ability to avoid unnecessary alarms.
Collapse
|
|
19
|
A cascaded spatiotemporal attention network for dynamic facial expression recognition. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03781-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
|
20
|
Liu Z, Pang L, Qi X. MEN: Mutual Enhancement Networks for Sign Language Recognition and Education. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022; PP:311-325. [PMID: 35613069 DOI: 10.1109/tnnls.2022.3174031] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The performance of existing sign language recognition approaches is typically limited by the scale of training data. To address this issue, we propose a mutual enhancement network (MEN) for joint sign language recognition and education. First, a sign language recognition system built upon a spatial-temporal network is proposed to recognize the semantic category of a given sign language video. Besides, a sign language education system is developed to detect the failure modes of learners and further guide them to sign correctly. Our theoretical contribution lies in formulating the above two systems as an estimation-maximization (EM) framework, which can progressively boost each other. The recognition system could become more robust and accurate with more training data collected by the education system, while the education system could guide the learners to sign more precisely, benefiting from the hand shape analysis module of the recognition system. Experimental results on three large-scale sign language recognition datasets validate the superiority of the proposed framework.
Collapse
|
|
21
|
Wu CL, Liu SF, Yu TL, Shih SJ, Chang CH, Yang Mao SF, Li YS, Chen HJ, Chen CC, Chao WC. Deep Learning-Based Pain Classifier Based on the Facial Expression in Critically Ill Patients. Front Med (Lausanne) 2022; 9:851690. [PMID: 35372435 PMCID: PMC8968070 DOI: 10.3389/fmed.2022.851690] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2022] [Accepted: 02/17/2022] [Indexed: 11/13/2022] Open
Abstract
ObjectivePain assessment based on facial expressions is an essential issue in critically ill patients, but an automated assessment tool is still lacking. We conducted this prospective study to establish the deep learning-based pain classifier based on facial expressions.MethodsWe enrolled critically ill patients during 2020–2021 at a tertiary hospital in central Taiwan and recorded video clips with labeled pain scores based on facial expressions, such as relaxed (0), tense (1), and grimacing (2). We established both image- and video-based pain classifiers through using convolutional neural network (CNN) models, such as Resnet34, VGG16, and InceptionV1 and bidirectional long short-term memory networks (BiLSTM). The performance of classifiers in the test dataset was determined by accuracy, sensitivity, and F1-score.ResultsA total of 63 participants with 746 video clips were eligible for analysis. The accuracy of using Resnet34 in the polychromous image-based classifier for pain scores 0, 1, 2 was merely 0.5589, and the accuracy of dichotomous pain classifiers between 0 vs. 1/2 and 0 vs. 2 were 0.7668 and 0.8593, respectively. Similar accuracy of image-based pain classifier was found using VGG16 and InceptionV1. The accuracy of the video-based pain classifier to classify 0 vs. 1/2 and 0 vs. 2 was approximately 0.81 and 0.88, respectively. We further tested the performance of established classifiers without reference, mimicking clinical scenarios with a new patient, and found the performance remained high.ConclusionsThe present study demonstrates the practical application of deep learning-based automated pain assessment in critically ill patients, and more studies are warranted to validate our findings.
Collapse
Affiliation(s)
- Chieh-Liang Wu
- Department of Critical Care Medicine, Taichung Veterans General Hospital, Taichung, Taiwan
- Department of Industrial Engineering and Enterprise Information, Tunghai University, Taichung, Taiwan
- Artificial Intelligence Studio, Taichung Veterans General Hospital, Taichung, Taiwan
- Colledge of Medicine, National Chung Hsing University, Taichung, Taiwan
| | - Shu-Fang Liu
- Department of Nursing, Taichung Veterans General Hospital, Taichung, Taiwan
| | - Tian-Li Yu
- Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
| | - Sou-Jen Shih
- Department of Nursing, Taichung Veterans General Hospital, Taichung, Taiwan
| | - Chih-Hung Chang
- Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
| | - Shih-Fang Yang Mao
- Electronic and Optoelectronic System Research Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan
| | - Yueh-Se Li
- Electronic and Optoelectronic System Research Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan
| | - Hui-Jiun Chen
- Department of Nursing, Taichung Veterans General Hospital, Taichung, Taiwan
| | - Chia-Chen Chen
- Electronic and Optoelectronic System Research Laboratories, Industrial Technology Research Institute, Hsinchu, Taiwan
- *Correspondence: Chia-Chen Chen
| | - Wen-Cheng Chao
- Department of Critical Care Medicine, Taichung Veterans General Hospital, Taichung, Taiwan
- Colledge of Medicine, National Chung Hsing University, Taichung, Taiwan
- Department of Automatic Control Engineering, Feng Chia University, Taichung, Taiwan
- Big Data Center, National Chung Hsing University, Taichung, Taiwan
- Wen-Cheng Chao
| |
Collapse
|
|
22
|
Computer aided pain detection and intensity estimation using compact CNN based fusion network. Appl Soft Comput 2021. [DOI: 10.1016/j.asoc.2021.107780] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
|
23
|
Wang H, Won D, Yoon SW. An adaptive neural architecture optimization model for retinal disorder diagnosis on 3D medical images. Appl Soft Comput 2021. [DOI: 10.1016/j.asoc.2021.107686] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|
|
24
|
Rathee N, Pahal S, Sheoran P. Pain detection from facial expressions using domain adaptation technique. Pattern Anal Appl 2021. [DOI: 10.1007/s10044-021-01025-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
|
25
|
Bhattacharyya A, Chatterjee S, Sen S, Sinitca A, Kaplun D, Sarkar R. A deep learning model for classifying human facial expressions from infrared thermal images. Sci Rep 2021; 11:20696. [PMID: 34667253 PMCID: PMC8526608 DOI: 10.1038/s41598-021-99998-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Accepted: 10/05/2021] [Indexed: 11/15/2022] Open
Abstract
The analysis of human facial expressions from the thermal images captured by the Infrared Thermal Imaging (IRTI) cameras has recently gained importance compared to images captured by the standard cameras using light having a wavelength in the visible spectrum. It is because infrared cameras work well in low-light conditions and also infrared spectrum captures thermal distribution that is very useful for building systems like Robot interaction systems, quantifying the cognitive responses from facial expressions, disease control, etc. In this paper, a deep learning model called IRFacExNet (InfraRed Facial Expression Network) has been proposed for facial expression recognition (FER) from infrared images. It utilizes two building blocks namely Residual unit and Transformation unit which extract dominant features from the input images specific to the expressions. The extracted features help to detect the emotion of the subjects in consideration accurately. The Snapshot ensemble technique is adopted with a Cosine annealing learning rate scheduler to improve the overall performance. The performance of the proposed model has been evaluated on a publicly available dataset, namely IRDatabase developed by RWTH Aachen University. The facial expressions present in the dataset are Fear, Anger, Contempt, Disgust, Happy, Neutral, Sad, and Surprise. The proposed model produces 88.43% recognition accuracy, better than some state-of-the-art methods considered here for comparison. Our model provides a robust framework for the detection of accurate expression in the absence of visible light.
Collapse
Affiliation(s)
| | - Somnath Chatterjee
- Computer Science and Engineering Department, Future Institute of Engineering and Management, Kolkata, India
| | - Shibaprasad Sen
- Computer Science and Technology Department, University of Engineering and Management, Kolkata, India
| | - Aleksandr Sinitca
- Department of Automation and Control Processes, Saint Petersburg Electrotechnical University "LETI", Saint Petersburg, Russia
| | - Dmitrii Kaplun
- Department of Automation and Control Processes, Saint Petersburg Electrotechnical University "LETI", Saint Petersburg, Russia.
| | - Ram Sarkar
- Department of Computer Science and Engineering, Jadavpur University, Kolkata, India
| |
Collapse
|
|
26
|
Xin X, Li X, Yang S, Lin X, Zheng X. Pain expression assessment based on a locality and identity aware network. IET IMAGE PROCESSING 2021; 15:2948-2958. [DOI: 10.1049/ipr2.12282] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Affiliation(s)
- Xuwu Xin
- The Second Affiliated Hospital of Shantou University Medical College Shantou China
| | - Xiaowu Li
- The Second Affiliated Hospital of Shantou University Medical College Shantou China
| | - Shengfu Yang
- The First Affiliated Hospital of Jinan University Guangzhou China
| | - Xiaoyan Lin
- The Second Affiliated Hospital of Shantou University Medical College Shantou China
| | - Xin Zheng
- Joint Shantou International Eye Center of Shantou University and The Chinese University of Hong Kong Shantou China
| |
Collapse
|
|
27
|
Guo Y, Wang L, Xiao Y, Lin Y. A Personalized Spatial-Temporal Cold Pain Intensity Estimation Model Based on Facial Expression. IEEE JOURNAL OF TRANSLATIONAL ENGINEERING IN HEALTH AND MEDICINE 2021; 9:4901008. [PMID: 34650836 PMCID: PMC8500272 DOI: 10.1109/jtehm.2021.3116867] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/24/2021] [Revised: 07/07/2021] [Accepted: 08/30/2021] [Indexed: 11/10/2022]
Abstract
OBJECTIVE Pain assessment is of great importance in both clinical research and patient care. Facial expression analysis is becoming a key part of pain detection because it is convenient, automatic, and real-time. The aim of this study is to present a cold pain intensity estimation experiment, investigate the importance of the spatial-temporal information on facial expression based cold pain, and study the performance of the personalized model as well as the generalized model. METHODS A cold pain experiment was carried out and facial expressions from 29 subjects were extracted. Three different architectures (Inception V3, VGG-LSTM, and Convolutional LSTM) were used to estimate three intensities of cold pain: No pain, Moderate pain, and Severe Pain. Architectures with Sequential information were compared with single-frame architecture, showing the importance of spatial-temporal information on pain estimation. The performances of the personalized model and the generalized model were also compared. RESULTS A mean F1 score of 79.48% was achieved using Convolutional LSTM based on the personalized model. CONCLUSION This study demonstrates the potential for the estimation of cold pain intensity from facial expression analysis and shows that the personalized spatial-temporal framework has better performance in cold pain intensity estimation. SIGNIFICANCE This cold pain intensity estimator could allow convenient, automatic, and real-time use to provide continuous objective pain intensity estimations of subjects and patients.
Collapse
Affiliation(s)
- Yikang Guo
- Intelligent Human-Machine Systems LabMechanical and Industrial Engineering DepartmentCollege of Engineering, Northeastern UniversityBostonMA02115USA
| | - Li Wang
- Intelligent Human-Machine Systems LabMechanical and Industrial Engineering DepartmentCollege of Engineering, Northeastern UniversityBostonMA02115USA
| | - Yan Xiao
- College of Nursing and Health InnovationUniversity of Texas at ArlingtonArlingtonTX76019USA
| | - Yingzi Lin
- Intelligent Human-Machine Systems LabMechanical and Industrial Engineering DepartmentCollege of Engineering, Northeastern UniversityBostonMA02115USA
| |
Collapse
|
|
28
|
Thiam P, Hihn H, Braun DA, Kestler HA, Schwenker F. Multi-Modal Pain Intensity Assessment Based on Physiological Signals: A Deep Learning Perspective. Front Physiol 2021; 12:720464. [PMID: 34539444 PMCID: PMC8440852 DOI: 10.3389/fphys.2021.720464] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2021] [Accepted: 07/30/2021] [Indexed: 11/13/2022] Open
Abstract
Traditional pain assessment approaches ranging from self-reporting methods, to observational scales, rely on the ability of an individual to accurately assess and successfully report observed or experienced pain episodes. Automatic pain assessment tools are therefore more than desirable in cases where this specific ability is negatively affected by various psycho-physiological dispositions, as well as distinct physical traits such as in the case of professional athletes, who usually have a higher pain tolerance as regular individuals. Hence, several approaches have been proposed during the past decades for the implementation of an autonomous and effective pain assessment system. These approaches range from more conventional supervised and semi-supervised learning techniques applied on a set of carefully hand-designed feature representations, to deep neural networks applied on preprocessed signals. Some of the most prominent advantages of deep neural networks are the ability to automatically learn relevant features, as well as the inherent adaptability of trained deep neural networks to related inference tasks. Yet, some significant drawbacks such as requiring large amounts of data to train deep models and over-fitting remain. Both of these problems are especially relevant in pain intensity assessment, where labeled data is scarce and generalization is of utmost importance. In the following work we address these shortcomings by introducing several novel multi-modal deep learning approaches (characterized by specific supervised, as well as self-supervised learning techniques) for the assessment of pain intensity based on measurable bio-physiological data. While the proposed supervised deep learning approach is able to attain state-of-the-art inference performances, our self-supervised approach is able to significantly improve the data efficiency of the proposed architecture by automatically generating physiological data and simultaneously performing a fine-tuning of the architecture, which has been previously trained on a significantly smaller amount of data.
Collapse
Affiliation(s)
- Patrick Thiam
- Institute of Medical Systems Biology, Ulm University, Ulm, Germany.,Institute of Neural Information Processing, Ulm University, Ulm, Germany
| | - Heinke Hihn
- Institute of Neural Information Processing, Ulm University, Ulm, Germany
| | - Daniel A Braun
- Institute of Neural Information Processing, Ulm University, Ulm, Germany
| | - Hans A Kestler
- Institute of Medical Systems Biology, Ulm University, Ulm, Germany
| | | |
Collapse
|
|
29
|
Wang S, Yuan Y, Zheng X, Lu X. Local and correlation attention learning for subtle facial expression recognition. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.07.120] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
|
30
|
Automatic Pain Estimation from Facial Expressions: A Comparative Analysis Using Off-the-Shelf CNN Architectures. ELECTRONICS 2021. [DOI: 10.3390/electronics10161926] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Automatic pain recognition from facial expressions is a challenging problem that has attracted a significant attention from the research community. This article provides a comprehensive analysis on the topic by comparing some popular and Off-the-Shell CNN (Convolutional Neural Network) architectures, including MobileNet, GoogleNet, ResNeXt-50, ResNet18, and DenseNet-161. We use these networks in two distinct modes: stand alone mode or feature extractor mode. In stand alone mode, the models (i.e., the networks) are used for directly estimating the pain. In feature extractor mode, the “values” of the middle layers are extracted and used as inputs to classifiers, such as SVR (Support Vector Regression) and RFR (Random Forest Regression). We perform extensive experiments on the benchmarking and publicly available database called UNBC-McMaster Shoulder Pain. The obtained results are interesting as they give valuable insights into the usefulness of the hidden CNN layers for automatic pain estimation.
Collapse
|
|
31
|
Deep transfer learning in human–robot interaction for cognitive and physical rehabilitation purposes. Pattern Anal Appl 2021. [DOI: 10.1007/s10044-021-00988-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
|
32
|
Hassan T, Seus D, Wollenberg J, Weitz K, Kunz M, Lautenbacher S, Garbas JU, Schmid U. Automatic Detection of Pain from Facial Expressions: A Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2021; 43:1815-1831. [PMID: 31825861 DOI: 10.1109/tpami.2019.2958341] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Pain sensation is essential for survival, since it draws attention to physical threat to the body. Pain assessment is usually done through self-reports. However, self-assessment of pain is not available in the case of noncommunicative patients, and therefore, observer reports should be relied upon. Observer reports of pain could be prone to errors due to subjective biases of observers. Moreover, continuous monitoring by humans is impractical. Therefore, automatic pain detection technology could be deployed to assist human caregivers and complement their service, thereby improving the quality of pain management, especially for noncommunicative patients. Facial expressions are a reliable indicator of pain, and are used in all observer-based pain assessment tools. Following the advancements in automatic facial expression analysis, computer vision researchers have tried to use this technology for developing approaches for automatically detecting pain from facial expressions. This paper surveys the literature published in this field over the past decade, categorizes it, and identifies future research directions. The survey covers the pain datasets used in the reviewed literature, the learning tasks targeted by the approaches, the features extracted from images and image sequences to represent pain-related information, and finally, the machine learning methods used.
Collapse
|
|
33
|
Andersen PH, Broomé S, Rashid M, Lundblad J, Ask K, Li Z, Hernlund E, Rhodin M, Kjellström H. Towards Machine Recognition of Facial Expressions of Pain in Horses. Animals (Basel) 2021; 11:1643. [PMID: 34206077 PMCID: PMC8229776 DOI: 10.3390/ani11061643] [Citation(s) in RCA: 25] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2021] [Revised: 05/26/2021] [Accepted: 05/28/2021] [Indexed: 01/30/2023] Open
Abstract
Automated recognition of human facial expressions of pain and emotions is to a certain degree a solved problem, using approaches based on computer vision and machine learning. However, the application of such methods to horses has proven difficult. Major barriers are the lack of sufficiently large, annotated databases for horses and difficulties in obtaining correct classifications of pain because horses are non-verbal. This review describes our work to overcome these barriers, using two different approaches. One involves the use of a manual, but relatively objective, classification system for facial activity (Facial Action Coding System), where data are analyzed for pain expressions after coding using machine learning principles. We have devised tools that can aid manual labeling by identifying the faces and facial keypoints of horses. This approach provides promising results in the automated recognition of facial action units from images. The second approach, recurrent neural network end-to-end learning, requires less extraction of features and representations from the video but instead depends on large volumes of video data with ground truth. Our preliminary results suggest clearly that dynamics are important for pain recognition and show that combinations of recurrent neural networks can classify experimental pain in a small number of horses better than human raters.
Collapse
Affiliation(s)
- Pia Haubro Andersen
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Sofia Broomé
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, SE 100044 Stockholm, Sweden; (S.B.); (Z.L.)
| | - Maheen Rashid
- Department of Computer Science, University of California at Davis, California, CA 95616, USA;
| | - Johan Lundblad
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Katrina Ask
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Zhenghong Li
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, SE 100044 Stockholm, Sweden; (S.B.); (Z.L.)
- Department of Computer Science, Stony Brook University, New York, NY 11794, USA
| | - Elin Hernlund
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Marie Rhodin
- Department of Anatomy, Physiology and Biochemistry, Swedish University of Agricultural Sciences, SE 75007 Uppsala, Sweden; (J.L.); (K.A.); (E.H.); (M.R.)
| | - Hedvig Kjellström
- Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, SE 100044 Stockholm, Sweden; (S.B.); (Z.L.)
| |
Collapse
|
|
34
|
Ralli AM, Chrysochoou E, Roussos P, Diakogiorgi K, Dimitropoulou P, Filippatou D. Executive Function, Working Memory, and Verbal Fluency in Relation to Non-Verbal Intelligence in Greek-Speaking School-Age Children with Developmental Language Disorder. Brain Sci 2021; 11:brainsci11050604. [PMID: 34066872 PMCID: PMC8151609 DOI: 10.3390/brainsci11050604] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2021] [Revised: 04/26/2021] [Accepted: 04/27/2021] [Indexed: 12/02/2022] Open
Abstract
Developmental Language Disorder (DLD) is often associated with impairments in working memory (WM), executive functions (EF), and verbal fluency. Moreover, increasing evidence shows poorer performance of children with DLD on non-verbal intelligence tests relative to their typically developing (TD) peers. Yet, the degree and generality of relevant difficulties remain unclear. The present study aimed at investigating WM capacity, key EFs and verbal fluency in relation to non-verbal intelligence in Greek-speaking school-age children with DLD, compared to TD peers (8–9 years). To our knowledge, the present study is the first to attempt a systematic relevant assessment with Greek-speaking school-age children, complementing previous studies mostly involving English-speaking participants. The results showed that children with DLD scored lower than TD peers on the non-verbal intelligence measure. Groups did not differ in the inhibition measures obtained (tapping resistance to either distractor or proactive interference), but children with DLD were outperformed by TD peers in the WM capacity, updating, monitoring (mixing cost), and verbal fluency (phonological and semantic) measures. The effects showed limited (in the case of backward digit recall) or no dependence on non-verbal intelligence. Findings are discussed in terms of their theoretical and practical implications as well as in relation to future lines of research.
Collapse
Affiliation(s)
- Asimina M. Ralli
- Department of Psychology, National and Kapodistrian University of Athens, 15784 Athens, Greece; (P.R.); (D.F.)
- Correspondence: ; Tel.: +30-210-7277945
| | - Elisavet Chrysochoou
- School of Psychology, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece;
| | - Petros Roussos
- Department of Psychology, National and Kapodistrian University of Athens, 15784 Athens, Greece; (P.R.); (D.F.)
| | | | | | - Diamanto Filippatou
- Department of Psychology, National and Kapodistrian University of Athens, 15784 Athens, Greece; (P.R.); (D.F.)
| |
Collapse
|
|
35
|
Xie W, Shen L, Duan J. Adaptive Weighting of Handcrafted Feature Losses for Facial Expression Recognition. IEEE TRANSACTIONS ON CYBERNETICS 2021; 51:2787-2800. [PMID: 31395570 DOI: 10.1109/tcyb.2019.2925095] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Due to the importance of facial expressions in human-machine interaction, a number of handcrafted features and deep neural networks have been developed for facial expression recognition. While a few studies have shown the similarity between the handcrafted features and the features learned by deep network, a new feature loss is proposed to use feature bias constraint of handcrafted and deep features to guide the deep feature learning during the early training of network. The feature maps learned with and without the proposed feature loss for a toy network suggest that our approach can fully explore the complementarity between handcrafted features and deep features. Based on the feature loss, a general framework for embedding the traditional feature information into deep network training was developed and tested using the FER2013, CK+, Oulu-CASIA, and MMI datasets. Moreover, adaptive loss weighting strategies are proposed to balance the influence of different losses for different expression databases. The experimental results show that the proposed feature loss with adaptive weighting achieves much better accuracy than the original handcrafted feature and the network trained without using our feature loss. Meanwhile, the feature loss with adaptive weighting can provide complementary information to compensate for the deficiency of a single feature.
Collapse
|
|
36
|
Semwal A, Londhe ND. MVFNet: A multi-view fusion network for pain intensity assessment in unconstrained environment. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102537] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
|
37
|
Hu Y, Wen G, Liao H, Wang C, Dai D, Yu Z. Automatic Construction of Chinese Herbal Prescriptions From Tongue Images Using CNNs and Auxiliary Latent Therapy Topics. IEEE TRANSACTIONS ON CYBERNETICS 2021; 51:708-721. [PMID: 31059462 DOI: 10.1109/tcyb.2019.2909925] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
The tongue image provides important physical information of humans. It is of great importance for diagnoses and treatments in clinical medicine. Herbal prescriptions are simple, noninvasive, and have low side effects. Thus, they are widely applied in China. Studies on the automatic construction technology of herbal prescriptions based on tongue images have great significance for deep learning to explore the relevance of tongue images for herbal prescriptions, it can be applied to healthcare services in mobile medical systems. In order to adapt to the tongue image in a variety of photographic environments and construct herbal prescriptions, a neural network framework for prescription construction is designed. It includes single/double convolution channels and fully connected layers. Furthermore, it proposes the auxiliary therapy topic loss mechanism to model the therapy of Chinese doctors and alleviate the interference of sparse output labels on the diversity of results. The experiment use the real-world tongue images and the corresponding prescriptions and the results can generate prescriptions that are close to the real samples, which verifies the feasibility of the proposed method for the automatic construction of herbal prescriptions from tongue images. Also, it provides a reference for automatic herbal prescription construction from more physical information.
Collapse
|
|
38
|
Salekin MS, Zamzmi G, Goldgof D, Kasturi R, Ho T, Sun Y. Multimodal spatio-temporal deep learning approach for neonatal postoperative pain assessment. Comput Biol Med 2021; 129:104150. [PMID: 33348218 PMCID: PMC7856028 DOI: 10.1016/j.compbiomed.2020.104150] [Citation(s) in RCA: 22] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2020] [Revised: 11/23/2020] [Accepted: 11/24/2020] [Indexed: 10/22/2022]
Abstract
The current practice for assessing neonatal postoperative pain relies on bedside caregivers. This practice is subjective, inconsistent, slow, and discontinuous. To develop a reliable medical interpretation, several automated approaches have been proposed to enhance the current practice. These approaches are unimodal and focus mainly on assessing neonatal procedural (acute) pain. As pain is a multimodal emotion that is often expressed through multiple modalities, the multimodal assessment of pain is necessary especially in case of postoperative (acute prolonged) pain. Additionally, spatio-temporal analysis is more stable over time and has been proven to be highly effective at minimizing misclassification errors. In this paper, we present a novel multimodal spatio-temporal approach that integrates visual and vocal signals and uses them for assessing neonatal postoperative pain. We conduct comprehensive experiments to investigate the effectiveness of the proposed approach. We compare the performance of the multimodal and unimodal postoperative pain assessment, and measure the impact of temporal information integration. The experimental results, on a real-world dataset, show that the proposed multimodal spatio-temporal approach achieves the highest AUC (0.87) and accuracy (79%), which are on average 6.67% and 6.33% higher than unimodal approaches. The results also show that the integration of temporal information markedly improves the performance as compared to the non-temporal approach as it captures changes in the pain dynamic. These results demonstrate that the proposed approach can be used as a viable alternative to manual assessment, which would tread a path toward fully automated pain monitoring in clinical settings, point-of-care testing, and homes.
Collapse
Affiliation(s)
- Md Sirajus Salekin
- Department of Computer Science and Engineering, University of South Florida, Tampa, FL, USA.
| | - Ghada Zamzmi
- Department of Computer Science and Engineering, University of South Florida, Tampa, FL, USA
| | - Dmitry Goldgof
- Department of Computer Science and Engineering, University of South Florida, Tampa, FL, USA
| | - Rangachar Kasturi
- Department of Computer Science and Engineering, University of South Florida, Tampa, FL, USA
| | - Thao Ho
- College of Medicine Pediatrics, USF Health, University of South Florida, Tampa, FL, USA
| | - Yu Sun
- Department of Computer Science and Engineering, University of South Florida, Tampa, FL, USA
| |
Collapse
|
|
39
|
Li Y, Ghosh S, Joshi J. PLAAN: Pain Level Assessment with Anomaly-detection based Network. JOURNAL ON MULTIMODAL USER INTERFACES 2021; 15:359-372. [PMCID: PMC7786324 DOI: 10.1007/s12193-020-00362-8] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/09/2020] [Accepted: 11/17/2020] [Indexed: 06/18/2023]
Abstract
Automatic chronic pain assessment and pain intensity estimation has been attracting growing attention due to its widespread applications. One of the prevalent issues in automatic pain analysis is inadequate balanced expert-labelled data for pain estimation. This work proposes an anomaly detection based network addressing one of the existing limitations of automatic pain assessment. The evaluation of the network is performed on pain intensity estimation and protective behaviour estimation tasks from body movements in the EmoPain Challenge dataset. The EmoPain dataset consists of body part based sensor data for both the tasks. The proposed network, PLAAN (Pain Level Assessment with Anomaly-detection based Network), is a lightweight LSTM-DNN network which considers features based on sensor data as the input and predicts intensity level of pain and presence or absence of protective behaviour in chronic low back pain patients. Joint training considering body movement patterns, such as exercise type, corresponding to pain exhibition as a label improves the performance of the network. However, contrary to perception, protective behaviour rather exists sporadically alongside pain in the EmoPain dataset. This induces yet another complication in accurate estimation of protective behaviour. This problem is resolved by incorporating anomaly detection in the network. A detailed comparison of different networks with varied features is outlined in the paper, presenting a significant improvement with the final proposed anomaly detection based network.
Collapse
Affiliation(s)
- Yi Li
- Human Centered AI at Monash University, Melbourne, Australia
| | - Shreya Ghosh
- Human Centered AI at Monash University, Melbourne, Australia
| | - Jyoti Joshi
- Human Centered AI at Monash University, Melbourne, Australia
| |
Collapse
|
|
40
|
Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation. Pattern Recognit Lett 2020. [DOI: 10.1016/j.patrec.2020.09.012] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
|
41
|
Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H. The modeling of human facial pain intensity based on Temporal Convolutional Networks trained with video frames in HSV color space. Appl Soft Comput 2020. [DOI: 10.1016/j.asoc.2020.106805] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
|
42
|
Performance Evaluation of Keyword Extraction Methods and Visualization for Student Online Comments. Symmetry (Basel) 2020. [DOI: 10.3390/sym12111923] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
Topic keyword extraction (as a typical task in information retrieval) refers to extracting the core keywords from document topics. In an online environment, students often post comments in subject forums. The automatic and accurate extraction of keywords from these comments are beneficial to lecturers (particular when it comes to repeatedly delivered subjects). In this paper, we compare the performance of traditional machine learning algorithms and two deep learning methods in extracting topic keywords from student comments posted in subject forums. For this purpose, we collected student comment data from a period of two years, manually tagging part of the raw data for our experiments. Based on this dataset, we comprehensively compared the five typical algorithms of naïve Bayes, logistic regression, support vector machine, convolutional neural networks, and Long Short-Term Memory with Attention (Att-LSTM). The performances were measured by the four evaluation metrics. We further examined the keywords by visualization. From the results of our experiment and visualization, we conclude that the Att-LSTM method is the best approach for topic keyword extraction from student comments. Further, the results from the algorithms and visualization are symmetry, to some degree. In particular, the extracted topics from the comments posted at the same stages of different teaching sessions are, almost, reflection symmetry.
Collapse
|
|
43
|
Erekat D, Hammal Z, Siddiqui M, Dibeklioğlu H. Enforcing Multilabel Consistency for Automatic Spatio-Temporal Assessment of Shoulder Pain Intensity. PROCEEDINGS OF THE ... ACM INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION. ICMI (CONFERENCE) 2020; 2020:156-164. [PMID: 34755152 PMCID: PMC8574156 DOI: 10.1145/3395035.3425190] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Abstract
The standard clinical assessment of pain is limited primarily to self-reported pain or clinician impression. While the self-reported measurement of pain is useful, in some circumstances it cannot be obtained. Automatic facial expression analysis has emerged as a potential solution for an objective, reliable, and valid measurement of pain. In this study, we propose a video based approach for the automatic measurement of self-reported pain and the observer pain intensity, respectively. To this end, we explore the added value of three self-reported pain scales, i.e., the Visual Analog Scale (VAS), the Sensory Scale (SEN), and the Affective Motivational Scale (AFF), as well as the Observer Pain Intensity (OPI) rating for a reliable assessment of pain intensity from facial expression. Using a spatio-temporal Convolutional Neural Network - Recurrent Neural Network (CNN-RNN) architecture, we propose to jointly minimize the mean absolute error of pain scores estimation for each of these scales while maximizing the consistency between them. The reliability of the proposed method is evaluated on the benchmark database for pain measurement from videos, namely, the UNBC-McMaster Pain Archive. Our results show that enforcing the consistency between different self-reported pain intensity scores collected using different pain scales enhances the quality of predictions and improve the state of the art in automatic self-reported pain estimation. The obtained results suggest that automatic assessment of self-reported pain intensity from videos is feasible, and could be used as a complementary instrument to unburden caregivers, specially for vulnerable populations that need constant monitoring.
Collapse
Affiliation(s)
- Diyala Erekat
- Department of Computer Engineering, Bilkent University, Ankara, Turkey
| | - Zakia Hammal
- The Robotics Institute, Carnegie Mellon University, Pittsburgh, USA
| | - Maimoon Siddiqui
- The Robotics Institute, Carnegie Mellon University, Pittsburgh, USA
| | - Hamdi Dibeklioğlu
- Department of Computer Engineering, Bilkent University, Ankara, Turkey
| |
Collapse
|
|
44
|
Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H. Ensemble neural network approach detecting pain intensity from facial expressions. Artif Intell Med 2020; 109:101954. [PMID: 34756219 DOI: 10.1016/j.artmed.2020.101954] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2020] [Revised: 08/23/2020] [Accepted: 08/25/2020] [Indexed: 11/28/2022]
Abstract
This paper reports on research to design an ensemble deep learning framework that integrates fine-tuned, three-stream hybrid deep neural network (i.e., Ensemble Deep Learning Model, EDLM), employing Convolutional Neural Network (CNN) to extract facial image features, detect and accurately classify the pain. To develop the approach, the VGGFace is fine-tuned and integrated with Principal Component Analysis and employed to extract features in images from the Multimodal Intensity Pain database at the early phase of the model fusion. Subsequently, a late fusion, three layers hybrid CNN and recurrent neural network algorithm is developed with their outputs merged to produce image-classified features to classify pain levels. The EDLM model is then benchmarked by means of a single-stream deep learning model including several competing models based on deep learning methods. The results obtained indicate that the proposed framework is able to outperform the competing methods, applied in a multi-level pain detection database to produce a feature classification accuracy that exceeds 89 %, with a receiver operating characteristic of 93 %. To evaluate the generalization of the proposed EDLM model, the UNBC-McMaster Shoulder Pain dataset is used as a test dataset for all of the modelling experiments, which reveals the efficacy of the proposed method for pain classification from facial images. The study concludes that the proposed EDLM model can accurately classify pain and generate multi-class pain levels for potential applications in the medical informatics area, and should therefore, be explored further in expert systems for detecting and classifying the pain intensity of patients, and automatically evaluating the patients' pain level accurately.
Collapse
Affiliation(s)
- Ghazal Bargshady
- School of Management and Enterprise, University of Southern Queensland, Springfield, Qld 4300, Australia.
| | - Xujuan Zhou
- School of Management and Enterprise, University of Southern Queensland, Springfield, Qld 4300, Australia.
| | - Ravinesh C Deo
- School of Sciences, University of Southern Queensland, Springfield, Qld 4300, Australia.
| | - Jeffrey Soar
- School of Management and Enterprise, University of Southern Queensland, Springfield, Qld 4300, Australia.
| | - Frank Whittaker
- School of Management and Enterprise, University of Southern Queensland, Springfield, Qld 4300, Australia.
| | - Hua Wang
- Victoria University, Melbourne, Australia.
| |
Collapse
|
|
45
|
Wang S, Zheng Z, Yin S, Yang J, Ji Q. A Novel Dynamic Model Capturing Spatial and Temporal Patterns for Facial Expression Analysis. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2020; 42:2082-2095. [PMID: 30998459 DOI: 10.1109/tpami.2019.2911937] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Facial expression analysis could be greatly improved by incorporating spatial and temporal patterns present in facial behavior, but the patterns have not yet been utilized to their full advantage. We remedy this via a novel dynamic model-an interval temporal restricted Boltzmann machine (IT-RBM) - that is able to capture both universal spatial patterns and complicated temporal patterns in facial behavior for facial expression analysis. We regard a facial expression as a multifarious activity composed of sequential or overlapping primitive facial events. Allen's interval algebra is implemented to portray these complicated temporal patterns via a two-layer Bayesian network. The nodes in the upper-most layer are representative of the primitive facial events, and the nodes in the lower layer depict the temporal relationships between those events. Our model also captures inherent universal spatial patterns via a multi-value restricted Boltzmann machine in which the visible nodes are facial events, and the connections between hidden and visible nodes model intrinsic spatial patterns. Efficient learning and inference algorithms are proposed. Experiments on posed and spontaneous expression distinction and expression recognition demonstrate that our proposed IT-RBM achieves superior performance compared to state-of-the art research due to its ability to incorporate these facial behavior patterns.
Collapse
|
|
46
|
Deep-Learning-Based Models for Pain Recognition: A Systematic Review. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10175984] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Traditional standards employed for pain assessment have many limitations. One such limitation is reliability linked to inter-observer variability. Therefore, there have been many approaches to automate the task of pain recognition. Recently, deep-learning methods have appeared to solve many challenges such as feature selection and cases with a small number of data sets. This study provides a systematic review of pain-recognition systems that are based on deep-learning models for the last two years. Furthermore, it presents the major deep-learning methods used in the review papers. Finally, it provides a discussion of the challenges and open issues.
Collapse
|
|
47
|
Xin X, Lin X, Yang S, Zheng X. Pain intensity estimation based on a spatial transformation and attention CNN. PLoS One 2020; 15:e0232412. [PMID: 32822348 PMCID: PMC7444520 DOI: 10.1371/journal.pone.0232412] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2019] [Accepted: 04/14/2020] [Indexed: 02/05/2023] Open
Abstract
Models designed to detect abnormalities that reflect disease from facial structures are an emerging area of research for automated facial analysis, which has important potential value in smart healthcare applications. However, most of the proposed models directly analyze the whole face image containing the background information, and rarely consider the effects of the background and different face regions on the analysis results. Therefore, in view of these effects, we propose an end-to-end attention network with spatial transformation to estimate different pain intensities. In the proposed method, the face image is first provided as input to a spatial transformation network for solving the problem of background interference; then, the attention mechanism is used to adaptively adjust the weights of different face regions of the transformed face image; finally, a convolutional neural network (CNN) containing a Softmax function is utilized to classify the pain levels. The extensive experiments and analysis are conducted on the benchmarking and publicly available database, namely the UNBC-McMaster shoulder pain. More specifically, in order to verify the superiority of our proposed method, the comparisons with the basic CNNs and the-state-of-the-arts are performed, respectively. The experiments show that the introduced spatial transformation and attention mechanism in our method can significantly improve the estimation performances and outperform the-state-of-the-arts.
Collapse
Affiliation(s)
- Xuwu Xin
- The Second Affiliated Hospital of Shantou University Medical College, Shantou, China
| | - Xiaoyan Lin
- The Second Affiliated Hospital of Shantou University Medical College, Shantou, China
| | - Shengfu Yang
- The First Affiliated Hospital of Jinan University, Guangzhou, China
| | - Xin Zheng
- Shantou Chaonan Minsheng Hospital, Shantou, China
| |
Collapse
|
|
48
|
Liu Y, Zhang X, Lin Y, Wang H. Facial Expression Recognition via Deep Action Units Graph Network Based on Psychological Mechanism. IEEE Trans Cogn Dev Syst 2020. [DOI: 10.1109/tcds.2019.2917711] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
|
49
|
Lustgarten JL, Zehnder A, Shipman W, Gancher E, Webb TL. Veterinary informatics: forging the future between veterinary medicine, human medicine, and One Health initiatives-a joint paper by the Association for Veterinary Informatics (AVI) and the CTSA One Health Alliance (COHA). JAMIA Open 2020; 3:306-317. [PMID: 32734172 PMCID: PMC7382640 DOI: 10.1093/jamiaopen/ooaa005] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2019] [Revised: 12/26/2019] [Accepted: 02/26/2020] [Indexed: 12/25/2022] Open
Abstract
Objectives This manuscript reviews the current state of veterinary medical electronic health records and the ability to aggregate and analyze large datasets from multiple organizations and clinics. We also review analytical techniques as well as research efforts into veterinary informatics with a focus on applications relevant to human and animal medicine. Our goal is to provide references and context for these resources so that researchers can identify resources of interest and translational opportunities to advance the field. Methods and Results This review covers various methods of veterinary informatics including natural language processing and machine learning techniques in brief and various ongoing and future projects. After detailing techniques and sources of data, we describe some of the challenges and opportunities within veterinary informatics as well as providing reviews of common One Health techniques and specific applications that affect both humans and animals. Discussion Current limitations in the field of veterinary informatics include limited sources of training data for developing machine learning and artificial intelligence algorithms, siloed data between academic institutions, corporate institutions, and many small private practices, and inconsistent data formats that make many integration problems difficult. Despite those limitations, there have been significant advancements in the field in the last few years and continued development of a few, key, large data resources that are available for interested clinicians and researchers. These real-world use cases and applications show current and significant future potential as veterinary informatics grows in importance. Veterinary informatics can forge new possibilities within veterinary medicine and between veterinary medicine, human medicine, and One Health initiatives.
Collapse
Affiliation(s)
- Jonathan L Lustgarten
- Association for Veterinary Informatics, Dixon, California, USA.,VCA Inc., Health Technology & Informatics, Los Angeles, California, USA
| | | | - Wayde Shipman
- Veterinary Medical Databases, Columbia, Missouri, USA
| | - Elizabeth Gancher
- Department of Infectious diseases and HIV medicine, Drexel University College of Medicine, Philadelphia, Pennsylvania, USA
| | - Tracy L Webb
- Department of Clinical Sciences, Colorado State University, Fort Collins, Colorado, USA
| |
Collapse
|
|
50
|
Wan Z, Jiang C, Fahad M, Ni Z, Guo Y, He H. Robot-Assisted Pedestrian Regulation Based on Deep Reinforcement Learning. IEEE TRANSACTIONS ON CYBERNETICS 2020; 50:1669-1682. [PMID: 30475740 DOI: 10.1109/tcyb.2018.2878977] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Pedestrian regulation can prevent crowd accidents and improve crowd safety in densely populated areas. Recent studies use mobile robots to regulate pedestrian flows for desired collective motion through the effect of passive human-robot interaction (HRI). This paper formulates a robot motion planning problem for the optimization of two merging pedestrian flows moving through a bottleneck exit. To address the challenge of feature representation of complex human motion dynamics under the effect of HRI, we propose using a deep neural network to model the mapping from the image input of pedestrian environments to the output of robot motion decisions. The robot motion planner is trained end-to-end using a deep reinforcement learning algorithm, which avoids hand-crafted feature detection and extraction, thus improving the learning capability for complex dynamic problems. Our proposed approach is validated in simulated experiments, and its performance is evaluated. The results demonstrate that the robot is able to find optimal motion decisions that maximize the pedestrian outflow in different flow conditions, and the pedestrian-accumulated outflow increases significantly compared to cases without robot regulation and with random robot motion.
Collapse
|