Verification of using virtual reality to evaluate deficiencies in cognitive function among patients with schizophrenia in the remission stage: a cross-sectional study

Background Schizophrenia is associated with widespread cognitive impairment. The MATRICS Consensus Cognitive Battery (MCCB) is most frequently used to assess cognitive function. However, the MCCB test is time consuming for the clinician. Virtual reality (VR) has emerged as an adjunctive tool to overcome this limitation and provides a new means to assess cognitive function. Methods The present study examined the validity and safety of using VR technology to assess cognitive function in Han Chinese patients with schizophrenia (SZs). The VR cognition training system (VRCTS) was used to simulate real-life supermarkets and assess cognitive function. Thirty-two SZs and 25 healthy controls (HCs) underwent VRCTS and MCCB assessments. An auxiliary diagnosis model was created based on the outcomes of the VRCTS to classify SZs and HCs by cognitive impairment. Results Significant differences in completion time between the SZs and HCs were detected using the VRCTS. SZs spent more time completing tasks than HCs. The outcome of VRCTS significantly correlated with the MCCB. The auxiliary diagnosis model had a sensitivity of 88.89% and a specificity of 88.89%. Conclusions These results support the use of VR technology in the assessment of cognitive impairment in Han Chinese schizophrenia patients. Trial registration China Clinical Trial Registry, ChiVTR1800016121. Registered 13 May 2018, http://www.chictr.org.cn/showproj.aspx?proj=27233


Background
Schizophrenia is a complex, heterogeneous behavioural and cognitive syndrome that is characterized by positive symptoms, negative symptoms and cognitive impairment [1,2]. Patients with schizophrenia (SZs) exhibit diminished cognitive function, including reduced attention and memory, and difficulties with executive functioning [3]. The MATRICS Consensus Cognitive Battery (MCCB), which includes 10 different cognitive subtests, is an accepted standard for the measurement of cognitive change in schizophrenia, and it is recommended by the United States Food and Drug Administration (FDA) for the assessment of cognitive impairment in schizophrenia [4,5]. The MCCB demonstrates excellent reliability and practicality. Recent studies showed that the MCCB was applicable for first-episode schizophrenia and chronic schizophrenia [6], and investigations using the MCCB focused on different cognitive domains in SZs. One study showed that the parents of SZs have conspicuous dysfunction in domains of working memory (WM), problem reasoning and visual learning (ViL) compared to parents of healthy controls (HCs) [7]. However, the MCCB requires a well-trained psychiatrist, and it takes approximately 1 h to evaluate cognitive function. The MCCB is also complicated for some SZs. Some patients may feel bored and exhausted to the point where they cannot complete the assessment. Therefore, it is necessary to find an easier and more attractive method to evaluate cognitive function.
Virtual reality (VR) has emerged as a tool to overcome this limitation of the MCCB and provide a new means to assess cognitive function. VR is a powerful tool that creates interactive computer-generated worlds that produce the sensation of being present in life-sized environments [8]. VR is already used in psychiatry. Current evidence suggests that the use of VR in SZs has great advantages. Some previous studies focused on the assessment of cognitive function using VR technology. A VR navigation task (VRNT) study reported that SZs showed significant impairment in memory compared to normal subjects [9]. Thirty-nine SZs and 21 HCs experienced a virtual maze, and SZs exhibited a higher rate of error [10]. Another study showed that SZs were deficient in life activities, medication management skills, and virtual character recognition [11][12][13].
Although studies reported the application of VR in psychiatry, there is limited literature on VR applications in Han Chinese people. A study in Hong Kong showed that VR provided a sensitive assessment of prospective memory deficits in SZs [14], but the efficiency and safety of VR technology are not sufficiently clear. The association between the VR method and the MCCB in the assessment of the cognitive function of SZs lacks evidence. Two studies in the United States demonstrated that the completion time of a VR task to assess functional capacity correlated with MCCB composite scores [15,16].
The present study examined VR technology performance for the assessment of cognitive function and the reliability of VR in distinguishing HCs and SZs in the remission stage in people of Han Chinese descent by cognitive function.

Methods
Participants 35 SZs in the Ningbo Psychiatric Hospital were recruited under the supervision of Dr. Sun, and 25 HC volunteers were recruited from the general population. All of the participants entered the VR environment, named the VR cognition training system (VRCTS), and underwent the MCCB test. The outcomes of VRCTS in the two groups were compared, and an auxiliary diagnosis model was created based on the outcomes of VRCTS to classify SZs and HCs using a support vector machine (SVM) method. Another 9 SZs and 9 HCs were recruited for verification of the auxiliary diagnosis model.
The following inclusion criteria were used: aged between 18 and 55 years; met the ICD-10 criteria for schizophrenia; Positive and Negative Syndrome Scale item scores of ≤3 or SAPS and SANS item scores of ≤2 for at least 6 months according to Andreasen's criteria, which reflects the remission stage [17]; only received atypical antipsychotics; normal vision; and right-handed. Exclusion criteria: a history of brain trauma, epilepsy and other neurological diseases or serious physical diseases; a history of mental retardation; a history of substance abuse in the past 30 days (except smoking); received electroconvulsive therapy in the past year; a history of using typical antipsychotics; and pregnancy or planning to become pregnant.
Three people in SZs declined because they could not understand the instructions of the MCCB and could not complete it. All SZs and HCs were matched by sex and age.
After describing the study to the subjects, written informed consent was obtained before the study was conducted, and subjects were offered an incentive of Ұ50 per session. The Ethics Committee of the First Affiliated Hospital of the Medical School of Zhejiang University approved the study, which was performed in accordance with the Helsinki Declaration (No. 2018533) and was previously registered in the China Clinical Trial Registry (No. ChiVTR1800016121).
Virtual reality procedure VRCTS was designed to assess cognitive function, performing different shopping tasks with different lists. The VRCTS simulated a supermarket with a variety of goods, such as drinks, tea sets, kitchen ware, fruits and vegetables. There was also a shopping cart in the simulation. Unity 5.3.5f1 (https://unity3d.com) and visual studio 2015 (Microsoft) were used to design and create the VRCTS.
The VRCTS included two tasks (task a and task b), and each task consisted of 4 different levels in total. Both tasks required participants to find goods and put them into shopping cart. Task a required participants to find goods of certain type, such as fruits, vegetables and drinks, and task b instructed participants to find specific goods, such as apples, tomatoes and cola. Participants need to finish task a and task b successively, from level 1 to level 4. The number of goods ranged from 3 to 6 as the task level increased from 1 to 4. The working memory (WM) span needed increased with the increasing number of goods. As a result, the different task levels represented different levels of difficulty.
Before each task started, there was a practice task that was used to bring all participants up to their best level of performance, and the outcome of the practice task was not included in statistical analyses. Participants became familiar with the procedures in the practice task as follows: The participants put on the helmet to begin the VRCT S task; A list of shopping goods appeared in the VR device, and the participant read the list and closed it after memorization; The participants chose items accordingly and put them in the shopping cart in the virtual supermarket using joysticks; If the participant forgot the contents of the list, he or she could press the button on the joysticks, and the list would appear again.
The participant was asked to finish task a and task b successively, using the same procedures as the practice task.
When the participant placed all of the goods from the list into the shopping cart, the computer software automatically recorded the correct number of items, errors, and completion time of each task and calculated the accuracy. The accuracy was equal to the correct number of items divided by the total number of goods in the cart.
If the accuracy was less than 100% the first time, the participant repeated the level. The second accuracy value was statistically analysed.
Completion time and accuracy were the major outcomes of the VR task that were used to evaluate cognitive function. The experiment did not limit the completion time of the task to allow each participant to obtain the highest possible accuracy values.
The VR situation was presented as follows (see Fig. 1).

Cognitive assessment
A trained psychiatrist assessed cognitive function using the MCCB. The MCCB includes 10 neurophysiologic tests that are clustered in 7 cognitive domains: speed of  [18]. Each domain score was standardized to a T score using the MCCB computer scoring program (Psychological Assessment Resources, Inc., version 2.1.1). The overall composite T score (CC) was calculated by averaging the standardized value of each test's T score.

Auxiliary diagnosis model and verification
We constructed an SVM-based cognitive impairment auxiliary diagnosis model for SZs in the remission stage based on outcomes from 57 subjects (32 SZs and 25 HCs), and outcomes from 18 additional subjects (9 SZs and 9 HCs) were analysed to verify the diagnostic effect.

Statistical analysis
The data are expressed as the mean ± SD for continuous variables. T-tests were used to compare age and education years of the SZs and HCs. The T scores of the MCCB and outcomes of VRCTS were analysed using covariance analysis (covariates: years of education

Characteristics of SZs and HCs
We However, SZs had fewer years of education than HCs (see Table 1).

Cognitive function of SZs and HCs
It took each participant approximately 1 h to finish the MCCB assessment. Covariance analysis (education) showed that SZs were significantly impaired in SP, AV, VeL, ViL, RPS, and SC compared to HCs (all Ps < 0.003) (see Table 2).

Comparison of performance on the VRCTS between SZs and HCs
The average total time taken to complete the VRCTS in SZs (1061±427 s) was significantly longer than the HCs (389±226 s). SZs spent significantly more time completing the different levels compared to HCs in task a level 1, level 2 and level 3 and task b level 2 and level 3 (Bonferroni correction, all Ps < 0.0125). However, there was no differences in accuracy of between SZs and HCs for any task (see Table 3).

Correlations of VRCTS outcome with the MCCB
For this analysis, we selected the completion time of each task level and the average completion time and correlated these values with the MCCB CC. These correlations are presented in Table 4 and Fig. 2. The average completion times of task a and task b significantly negatively correlated with the MCCB CC in SZs (Bonferroni correction, P< 0.0125). However, there was no similar correlation between the outcomes of the VRCTS and MCCB CC in HCs. We also analysed the average completion times of task a and task b with each MCCB domain. The results showed that the outcomes of VRCTS negatively   Table 5). The use of VRCTS as an auxiliary method to identify cognitive impairment.
Our results showed that the cognitive impairment auxiliary diagnosis model correctly classified SZs and HCs based on cognitive impairment. The completion time of VRCTS classified individuals based on cognitive impairment with 92.98% accuracy, 90.63% sensitivity, and 96% specificity. The area under the receiver operating characteristic (ROC) curve was 0.9325 (see Fig. 3). Verification based on another 18 subjects showed 88.89% accuracy, 88.89% sensitivity, 88.89% specificity, 88.89% positive predictive value and 88.89% negative predictive value.

Discussion
A virtual situation called VRCTS was constructed for the present study, and it simulated a real supermarket to assess cognitive function and examine the validity and safety of VR technology in Han Chinese SZs in the remission stage. SZs required nearly 18 min to complete the VRCTS compared to 6 min for HCs, which is  significantly shorter than the time required to complete the MCCB. The VRCTS was divided into two tasks, and each task consisted of 4 levels that required an increasing WM span. Task a was used to evaluate the processing of information related to certain categories, and task b was used to assess the processing of specific information. We designed different levels for each task, such as task a level 1 and level 2. The number of goods participants needed to find ranged from 3 to 6 as the task level increased. Two variables were used to measure the outcomes: accuracy of performance and the time to complete the task. Because there were no differences in task accuracy between SZs and HCs, it was meaningful to compare the completion times of the two groups.
Our results showed that the completion times for task a and task b of SZs were significantly higher than the completion times of HCs, which was likely due to the cognitive impairment of SZs. These findings are consistent with some studies. Thirty-three SZs and 39 HCs performed 10 trials of a virtual radial arm maze task, and the results showed that SZs took more time to finish the task compared to HCs [19]. Other studies also showed that SZs performed worse than HCs in the Virtual Action Planning-Supermarket (VAP-S) study and a Virtual Reality Functional Skills Assessment (VRFAS) [20,21]. The use of VR may assess the severity of theory of mind (ToM) impairment in SZs [22].  To finish the task of VRCTS, participants needed to read the list, find the goods on the list and put the goods into a shopping cart. Many cognitive domains are involved in these processes, such as SP, AV, WM and VeL. There were also significant negative correlations between outcome of VRCTS and the MCCB composite score and some domains, like SP, AV, WM and VeL. These correlations support our hypothesis that the VRCTS outcome reflects the cognitive impairment of SZs in the remission stage and primarily reflects SP, AV, WM and VeL.
Our results showed that the VRCTS distinguished SZs in the remission stage and HCs with high accuracy (88.89%), sensitivity (88.89%) and specificity (88.89%). This result means that SZs in the remission stage may be separated from healthy people based on their performance in the VRCTS. This result is consistent with a study of a VR prospective memory test, which demonstrated that the VR test examined prospective memory deficits in SZs with high sensitivity (92.9%) and specificity (75%) [14].
In summary, VR provides participants a feeling of presence that is similar to real life [23,24]. The present study demonstrated that VR technology, such as the VRCTS, was a time-saving, efficient and attractive method to evaluate cognitive function. Our model constructed a virtual situation that was applicable to Han Chinese people to evaluate cognitive function, and its outcome significantly correlated with the MCCB. The VRCTS helped distinguish SZs in the remission stage and healthy people based on cognitive impairment, which might support its use as a new adjunctive examination to evaluate cognitive impairment in SZs in the remission stage, especially people of Han Chinese descent. Although more research is needed, VR may be an attractive, safe and convenient auxiliary method for assessments in schizophrenics in the future.
This study had some limitations. First, the sample size was small. Second, we did not ask participants to assess their preference for MCCB vs. VRCTS. Therefore, this study lacks feedback from participants after they finished the VRCTS. As a result, it is difficult to rate the satisfaction and pleasure that the participants experienced. The SZs received different types of second-generation antipsychotics, which may interfere with the patients' cognitive function. As a result, further studies with SZs using a single second-generation antipsychotic are need to exclude the effects of antipsychotics on cognitive function. Third, the two groups were not education matched, and education may influence participants' performance in a virtual supermarket. Therefore, we used covariance analysis (covariates: years of education) to compare the outcome between SZs and HCs. Furthermore, our conclusion may not applied to first-onset, medication-naïve patients with less obvious cognitive impairment because the patients in our research were all SZs in the remission stage.

Conclusions
The VRCTS is a highly sensitive measure of the cognitive functions associated with the MCCB test. These results might support the use of VR technology in the assessment of cognitive function in SZs in the remission stage in Han Chinese patients.