Verification of using virtual reality to evaluate deficiencies in cognitive function among patients with schizophrenia in the remission stage: a cross-sectional study
BMC Psychiatry volume 21, Article number: 21 (2021)
Schizophrenia is associated with widespread cognitive impairment. The MATRICS Consensus Cognitive Battery (MCCB) is most frequently used to assess cognitive function. However, the MCCB test is time consuming for the clinician. Virtual reality (VR) has emerged as an adjunctive tool to overcome this limitation and provides a new means to assess cognitive function.
The present study examined the validity and safety of using VR technology to assess cognitive function in Han Chinese patients with schizophrenia (SZs). The VR cognition training system (VRCTS) was used to simulate real-life supermarkets and assess cognitive function. Thirty-two SZs and 25 healthy controls (HCs) underwent VRCTS and MCCB assessments. An auxiliary diagnosis model was created based on the outcomes of the VRCTS to classify SZs and HCs by cognitive impairment.
Significant differences in completion time between the SZs and HCs were detected using the VRCTS. SZs spent more time completing tasks than HCs. The outcome of VRCTS significantly correlated with the MCCB. The auxiliary diagnosis model had a sensitivity of 88.89% and a specificity of 88.89%.
These results support the use of VR technology in the assessment of cognitive impairment in Han Chinese schizophrenia patients.
China Clinical Trial Registry, ChiVTR1800016121. Registered 13 May 2018, http://www.chictr.org.cn/showproj.aspx?proj=27233
Schizophrenia is a complex, heterogeneous behavioural and cognitive syndrome that is characterized by positive symptoms, negative symptoms and cognitive impairment [1, 2]. Patients with schizophrenia (SZs) exhibit diminished cognitive function, including reduced attention and memory, and difficulties with executive functioning . The MATRICS Consensus Cognitive Battery (MCCB), which includes 10 different cognitive subtests, is an accepted standard for the measurement of cognitive change in schizophrenia, and it is recommended by the United States Food and Drug Administration (FDA) for the assessment of cognitive impairment in schizophrenia [4, 5]. The MCCB demonstrates excellent reliability and practicality. Recent studies showed that the MCCB was applicable for first-episode schizophrenia and chronic schizophrenia , and investigations using the MCCB focused on different cognitive domains in SZs. One study showed that the parents of SZs have conspicuous dysfunction in domains of working memory (WM), problem reasoning and visual learning (ViL) compared to parents of healthy controls (HCs) . However, the MCCB requires a well-trained psychiatrist, and it takes approximately 1 h to evaluate cognitive function. The MCCB is also complicated for some SZs. Some patients may feel bored and exhausted to the point where they cannot complete the assessment. Therefore, it is necessary to find an easier and more attractive method to evaluate cognitive function.
Virtual reality (VR) has emerged as a tool to overcome this limitation of the MCCB and provide a new means to assess cognitive function. VR is a powerful tool that creates interactive computer-generated worlds that produce the sensation of being present in life-sized environments . VR is already used in psychiatry. Current evidence suggests that the use of VR in SZs has great advantages. Some previous studies focused on the assessment of cognitive function using VR technology. A VR navigation task (VRNT) study reported that SZs showed significant impairment in memory compared to normal subjects . Thirty-nine SZs and 21 HCs experienced a virtual maze, and SZs exhibited a higher rate of error . Another study showed that SZs were deficient in life activities, medication management skills, and virtual character recognition [11,12,13].
Although studies reported the application of VR in psychiatry, there is limited literature on VR applications in Han Chinese people. A study in Hong Kong showed that VR provided a sensitive assessment of prospective memory deficits in SZs , but the efficiency and safety of VR technology are not sufficiently clear. The association between the VR method and the MCCB in the assessment of the cognitive function of SZs lacks evidence. Two studies in the United States demonstrated that the completion time of a VR task to assess functional capacity correlated with MCCB composite scores [15, 16].
The present study examined VR technology performance for the assessment of cognitive function and the reliability of VR in distinguishing HCs and SZs in the remission stage in people of Han Chinese descent by cognitive function.
35 SZs in the Ningbo Psychiatric Hospital were recruited under the supervision of Dr. Sun, and 25 HC volunteers were recruited from the general population. All of the participants entered the VR environment, named the VR cognition training system (VRCTS), and underwent the MCCB test. The outcomes of VRCTS in the two groups were compared, and an auxiliary diagnosis model was created based on the outcomes of VRCTS to classify SZs and HCs using a support vector machine (SVM) method. Another 9 SZs and 9 HCs were recruited for verification of the auxiliary diagnosis model.
The following inclusion criteria were used: aged between 18 and 55 years; met the ICD-10 criteria for schizophrenia; Positive and Negative Syndrome Scale item scores of ≤3 or SAPS and SANS item scores of ≤2 for at least 6 months according to Andreasen’s criteria, which reflects the remission stage ; only received atypical antipsychotics; normal vision; and right-handed. Exclusion criteria: a history of brain trauma, epilepsy and other neurological diseases or serious physical diseases; a history of mental retardation; a history of substance abuse in the past 30 days (except smoking); received electroconvulsive therapy in the past year; a history of using typical antipsychotics; and pregnancy or planning to become pregnant.
Three people in SZs declined because they could not understand the instructions of the MCCB and could not complete it. All SZs and HCs were matched by sex and age.
After describing the study to the subjects, written informed consent was obtained before the study was conducted, and subjects were offered an incentive of Ұ50 per session. The Ethics Committee of the First Affiliated Hospital of the Medical School of Zhejiang University approved the study, which was performed in accordance with the Helsinki Declaration (No. 2018533) and was previously registered in the China Clinical Trial Registry (No. ChiVTR1800016121).
Virtual reality procedure
VRCTS was designed to assess cognitive function, performing different shopping tasks with different lists. The VRCTS simulated a supermarket with a variety of goods, such as drinks, tea sets, kitchen ware, fruits and vegetables. There was also a shopping cart in the simulation. Unity 5.3.5f1 (https://unity3d.com) and visual studio 2015 (Microsoft) were used to design and create the VRCTS.
The VRCTS included two tasks (task a and task b), and each task consisted of 4 different levels in total. Both tasks required participants to find goods and put them into shopping cart. Task a required participants to find goods of certain type, such as fruits, vegetables and drinks, and task b instructed participants to find specific goods, such as apples, tomatoes and cola. Participants need to finish task a and task b successively, from level 1 to level 4. The number of goods ranged from 3 to 6 as the task level increased from 1 to 4. The working memory (WM) span needed increased with the increasing number of goods. As a result, the different task levels represented different levels of difficulty.
Before each task started, there was a practice task that was used to bring all participants up to their best level of performance, and the outcome of the practice task was not included in statistical analyses. Participants became familiar with the procedures in the practice task as follows:
The participants put on the helmet to begin the VRCTS task;
A list of shopping goods appeared in the VR device, and the participant read the list and closed it after memorization;
The participants chose items accordingly and put them in the shopping cart in the virtual supermarket using joysticks;
If the participant forgot the contents of the list, he or she could press the button on the joysticks, and the list would appear again.
The participant was asked to finish task a and task b successively, using the same procedures as the practice task.
When the participant placed all of the goods from the list into the shopping cart, the computer software automatically recorded the correct number of items, errors, and completion time of each task and calculated the accuracy. The accuracy was equal to the correct number of items divided by the total number of goods in the cart.
If the accuracy was less than 100% the first time, the participant repeated the level. The second accuracy value was statistically analysed.
Completion time and accuracy were the major outcomes of the VR task that were used to evaluate cognitive function. The experiment did not limit the completion time of the task to allow each participant to obtain the highest possible accuracy values.
The VR situation was presented as follows (see Fig. 1).
A trained psychiatrist assessed cognitive function using the MCCB. The MCCB includes 10 neurophysiologic tests that are clustered in 7 cognitive domains: speed of processing (SP), attention/vigilance (AV), WM, verbal learning (VeL), ViL, reasoning/problem solving (RPS), and social cognition (SC) . Each domain score was standardized to a T score using the MCCB computer scoring program (Psychological Assessment Resources, Inc., version 2.1.1). The overall composite T score (CC) was calculated by averaging the standardized value of each test’s T score.
Auxiliary diagnosis model and verification
We constructed an SVM-based cognitive impairment auxiliary diagnosis model for SZs in the remission stage based on outcomes from 57 subjects (32 SZs and 25 HCs), and outcomes from 18 additional subjects (9 SZs and 9 HCs) were analysed to verify the diagnostic effect.
The data are expressed as the mean ± SD for continuous variables. T-tests were used to compare age and education years of the SZs and HCs. The T scores of the MCCB and outcomes of VRCTS were analysed using covariance analysis (covariates: years of education). Correlation of T scores of the MCCB and outcomes of VRCTS was analysed using Pearson correlation. Statistical analyses were performed using SPSS version 19.0 (IBM, Chicago, IL, USA) for Windows. For verification of the diagnostic model, DTREG (https://www.dtreg.com/) was used to test the diagnostic model.
Characteristics of SZs and HCs
We recruited 35 SZs in the remission stage and 25 HCs. Data from 3 of the 35 SZs were excluded because the participants could not finish the MCCB. 32 SZs and 25 HCs completed the VRCTS and the MCCB. 5 SZs and 4 HCs felt dizzy during the VR tasks, but they finished all tasks because the dizziness was tolerable. No other uncomfortable feelings were reported. The age of SZs ranged from 24 to 54 years, and the age of HCs ranged from 28 to 51 years (mean age = 38.84 years SD=5.56). Subject demographic and clinical characteristics are provided in Table 1. The disease duration of SZs ranged from 12 to 384 months. SZs received second-generation antipsychotics for treatment. There were no significant differences in age or sex between groups (all Ps > 0.05). However, SZs had fewer years of education than HCs (see Table 1).
Cognitive function of SZs and HCs
It took each participant approximately 1 h to finish the MCCB assessment. Covariance analysis (education) showed that SZs were significantly impaired in SP, AV, VeL, ViL, RPS, and SC compared to HCs (all Ps < 0.003) (see Table 2).
Comparison of performance on the VRCTS between SZs and HCs
The average total time taken to complete the VRCTS in SZs (1061±427 s) was significantly longer than the HCs (389±226 s). SZs spent significantly more time completing the different levels compared to HCs in task a level 1, level 2 and level 3 and task b level 2 and level 3 (Bonferroni correction, all Ps < 0.0125). However, there was no differences in accuracy of between SZs and HCs for any task (see Table 3).
Correlations of VRCTS outcome with the MCCB
For this analysis, we selected the completion time of each task level and the average completion time and correlated these values with the MCCB CC. These correlations are presented in Table 4 and Fig. 2. The average completion times of task a and task b significantly negatively correlated with the MCCB CC in SZs (Bonferroni correction, P< 0.0125). However, there was no similar correlation between the outcomes of the VRCTS and MCCB CC in HCs.
We also analysed the average completion times of task a and task b with each MCCB domain. The results showed that the outcomes of VRCTS negatively correlated with SP, AV, WM and VeL (all Ps < 0.05). But there is no significant correlation in HCs (see Table 5).
The use of VRCTS as an auxiliary method to identify cognitive impairment.
Our results showed that the cognitive impairment auxiliary diagnosis model correctly classified SZs and HCs based on cognitive impairment. The completion time of VRCTS classified individuals based on cognitive impairment with 92.98% accuracy, 90.63% sensitivity, and 96% specificity. The area under the receiver operating characteristic (ROC) curve was 0.9325 (see Fig. 3). Verification based on another 18 subjects showed 88.89% accuracy, 88.89% sensitivity, 88.89% specificity, 88.89% positive predictive value and 88.89% negative predictive value.
A virtual situation called VRCTS was constructed for the present study, and it simulated a real supermarket to assess cognitive function and examine the validity and safety of VR technology in Han Chinese SZs in the remission stage. SZs required nearly 18 min to complete the VRCTS compared to 6 min for HCs, which is significantly shorter than the time required to complete the MCCB.
The VRCTS was divided into two tasks, and each task consisted of 4 levels that required an increasing WM span. Task a was used to evaluate the processing of information related to certain categories, and task b was used to assess the processing of specific information. We designed different levels for each task, such as task a level 1 and level 2. The number of goods participants needed to find ranged from 3 to 6 as the task level increased. Two variables were used to measure the outcomes: accuracy of performance and the time to complete the task. Because there were no differences in task accuracy between SZs and HCs, it was meaningful to compare the completion times of the two groups.
Our results showed that the completion times for task a and task b of SZs were significantly higher than the completion times of HCs, which was likely due to the cognitive impairment of SZs. These findings are consistent with some studies. Thirty-three SZs and 39 HCs performed 10 trials of a virtual radial arm maze task, and the results showed that SZs took more time to finish the task compared to HCs . Other studies also showed that SZs performed worse than HCs in the Virtual Action Planning-Supermarket (VAP-S) study and a Virtual Reality Functional Skills Assessment (VRFAS) [20, 21]. The use of VR may assess the severity of theory of mind (ToM) impairment in SZs .
To finish the task of VRCTS, participants needed to read the list, find the goods on the list and put the goods into a shopping cart. Many cognitive domains are involved in these processes, such as SP, AV, WM and VeL. There were also significant negative correlations between outcome of VRCTS and the MCCB composite score and some domains, like SP, AV, WM and VeL. These correlations support our hypothesis that the VRCTS outcome reflects the cognitive impairment of SZs in the remission stage and primarily reflects SP, AV, WM and VeL.
Our results showed that the VRCTS distinguished SZs in the remission stage and HCs with high accuracy (88.89%), sensitivity (88.89%) and specificity (88.89%). This result means that SZs in the remission stage may be separated from healthy people based on their performance in the VRCTS. This result is consistent with a study of a VR prospective memory test, which demonstrated that the VR test examined prospective memory deficits in SZs with high sensitivity (92.9%) and specificity (75%) .
In summary, VR provides participants a feeling of presence that is similar to real life [23, 24]. The present study demonstrated that VR technology, such as the VRCTS, was a time-saving, efficient and attractive method to evaluate cognitive function. Our model constructed a virtual situation that was applicable to Han Chinese people to evaluate cognitive function, and its outcome significantly correlated with the MCCB. The VRCTS helped distinguish SZs in the remission stage and healthy people based on cognitive impairment, which might support its use as a new adjunctive examination to evaluate cognitive impairment in SZs in the remission stage, especially people of Han Chinese descent. Although more research is needed, VR may be an attractive, safe and convenient auxiliary method for assessments in schizophrenics in the future.
This study had some limitations. First, the sample size was small. Second, we did not ask participants to assess their preference for MCCB vs. VRCTS. Therefore, this study lacks feedback from participants after they finished the VRCTS. As a result, it is difficult to rate the satisfaction and pleasure that the participants experienced. The SZs received different types of second-generation antipsychotics, which may interfere with the patients’ cognitive function. As a result, further studies with SZs using a single second-generation antipsychotic are need to exclude the effects of antipsychotics on cognitive function. Third, the two groups were not education matched, and education may influence participants’ performance in a virtual supermarket. Therefore, we used covariance analysis (covariates: years of education) to compare the outcome between SZs and HCs. Furthermore, our conclusion may not applied to first-onset, medication-naïve patients with less obvious cognitive impairment because the patients in our research were all SZs in the remission stage.
The VRCTS is a highly sensitive measure of the cognitive functions associated with the MCCB test. These results might support the use of VR technology in the assessment of cognitive function in SZs in the remission stage in Han Chinese patients.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
MATRICS Consensus Cognitive Battery
Patients with schizophrenia
VR cognition training system
Composite T score
VR navigation task
Support vector machine
Speed of processing
Virtual Action Planning-Supermarket
Virtual Reality Functional Skills Assessment
Owen MJ, Sawa A, Mortensen PB. Schizophrenia. Lancet. 2016;388(10039):86–97.
Joyce EM, Roiser JP. Cognitive heterogeneity in schizophrenia. Curr Opin Psychiatry. 2007;20(3):268–72.
Fioravanti M, Bianchi V, Cinti ME. Cognitive deficits in schizophrenia: an updated metanalysis of the scientific evidence. BMC Psychiatry. 2012;12:64.
Shi C, Kang L, Yao S, Ma Y, Li T, Liang Y, Cheng Z, Xu Y, Shi J, Xu X, et al. The MATRICS consensus cognitive battery (MCCB): co-norming and standardization in China. Schizophr Res. 2015;169(1–3):109–15.
Nuechterlein KH, Green MF, Kern RS, Baade LE, Barch DM, Cohen JD, Essock S, Fenton WS, Frese FJ 3rd, Gold JM, et al. The MATRICS consensus cognitive battery, part 1: test selection, reliability, and validity. Am J Psychiatry. 2008;165(2):203–13.
McCleery A, Ventura J, Kern RS, Subotnik KL, Gretchen-Doorly D, Green MF, Hellemann GS, Nuechterlein KH. Cognitive functioning in first-episode schizophrenia: MATRICS consensus cognitive battery (MCCB) profile of impairment. Schizophr Res. 2014;157(1–3):33–9.
Cao A, Shen T, Li H, Wu C, McCabe M, Mellor D, Byrne L, Zhang J, Huang J, Peng D, et al. Dysfunction of cognition patterns measured by MATRICS consensus cognitive battery (MCCB) among first episode schizophrenia patients and their biological parents. Shanghai Arch Psychiatry. 2017;29(3):154–60.
Freeman D, Reeve S, Robinson A, Ehlers A, Clark D, Spanlang B, Slater M. Virtual reality in the assessment, understanding, and treatment of mental health disorders. Psychol Med. 2017;47(14):2393–400.
Mohammadi A, Hesami E, Kargar M, Shams J. Detecting allocentric and egocentric navigation deficits in patients with schizophrenia and bipolar disorder using virtual reality. Neuropsychol Rehabil. 2018;28(3):398–415.
Sorkin A, Weinshall D, Modai I, Peled A. Improving the accuracy of the diagnosis of schizophrenia by means of virtual reality. Am J Psychiatry. 2006;163(3):512–20.
Kurtz MM, Baker E, Pearlson GD, Astur RS. A virtual reality apartment as a measure of medication management skills in patients with schizophrenia: a pilot study. Schizophr Bull. 2007;33(5):1162–70.
Baker EK, Kurtz MM, Astur RS. Virtual reality assessment of medication compliance in patients with schizophrenia. CyberPsychol Behav. 2006;9(2):224–9.
Dyck M, Winbeck M, Leiberg S, Chen Y, Mathiak K. Virtual faces as a tool to study emotion recognition deficits in schizophrenia. Psychiatry Res. 2010;179(3):247–52.
Man DWK, Ganesan B, Yip CCK, Lee COP, Tsang SYL, Yu PWP, Young JGE, Shum DHK. Validation of the virtual-reality prospective memory test (Hong Kong Chinese version) for individuals with first-episode schizophrenia. Neuropsychol Rehabil. 2018;28(7):1197–210.
Keefe RSE, Davis VG, Atkins AS, Vaughan A, Patterson T, Narasimhan M, Harvey PD. Validation of a computerized test of functional capacity. Schizophr Res. 2016;175(1–3):90–6.
Ruse SA, Harvey PD, Davis VG, Atkins AS, Fox KH, Keefe RS. Virtual reality functional capacity assessment in schizophrenia: preliminary data regarding feasibility and correlations with cognitive and functional capacity performance. Schizophr Res Cogn. 2014;1(1):e21–6.
Andreasen NC, Carpenter WT Jr, Kane JM, Lasser RA, Marder SR, Weinberger DR. Remission in schizophrenia: proposed criteria and rationale for consensus. Am J Psychiatry. 2005;162(3):441–9.
Sui J, Pearlson GD, Du Y, Yu Q, Jones TR, Chen J, Jiang T, Bustillo J, Calhoun VD. In search of multimodal neuroimaging biomarkers of cognitive deficits in schizophrenia. Biol Psychiatry. 2015;78(11):794–804.
Spieker EA, Astur RS, West JT, Griego JA, Rowland LM. Spatial memory deficits in a virtual reality eight-arm radial maze in schizophrenia. Schizophr Res. 2012;135(1–3):84–9.
Park KM, Ku J, Park IH, Park JY, Kim SI, Kim JJ. Improvement in social competence in patients with schizophrenia: a pilot study using a performance-based measure using virtual reality. Hum Psychopharmacol. 2009;24(8):619–27.
Josman N, Schenirderman AE, Klinger E, Shevil E. Using virtual reality to evaluate executive functioning among persons with schizophrenia: a validity study. Schizophr Res. 2009;115(2–3):270–7.
Canty AL, Neumann DL, Shum DHK. Using virtual reality to assess theory of mind subprocesses and error types in early and chronic schizophrenia. Schizophr Res Cogn. 2017;10:15–9.
Liu Y, Tan W, Chen C, Liu C, Yang J, Zhang Y. A review of the application of virtual reality Technology in the Diagnosis and Treatment of cognitive impairment. Front Aging Neurosci. 2019;11:280.
Park KM, Ku J, Choi SH, Jang HJ, Park JY, Kim SI, Kim JJ. A virtual reality application in role-plays of social skills training for schizophrenia: a randomized, controlled trial. Psychiatry Res. 2011;189(2):166–72.
We thank the patients for their involvement in the experiment and for their time and effort devoted to the consent process and interviews. We also thank the staff of the Ningbo Psychiatric Hospital for their assistance.
The design of the study and collection of data was supported by two grants, 2016YFC1307005 and 2016YFC1306900 from the National Key Research and Development Program of China, grant LGF18H090003 from the Basic Public Welfare Research projects in Zhejiang Province, grant 2015C03054 from the Major Subject of Zhejiang Province and grant 2016C37076 from the Zhejiang Provincial Science and Technology Department. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.
Ethics approval and consent to participate
All participants provided written informed consent prior to participation in the study.
The study was approved by the Ethics Committee of the First Affiliated Hospital of the Medical School of Zhejiang University in accordance with the Declaration of Helsinki.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Huang, B., Li, S., Sun, B. et al. Verification of using virtual reality to evaluate deficiencies in cognitive function among patients with schizophrenia in the remission stage: a cross-sectional study. BMC Psychiatry 21, 21 (2021). https://doi.org/10.1186/s12888-020-03029-6