Statistical validation of the criteria for symptom remission in schizophrenia: Preliminary findings
© Opler et al; licensee BioMed Central Ltd. 2007
Received: 05 February 2007
Accepted: 24 July 2007
Published: 24 July 2007
Published methods for assessing remission in schizophrenia are variable and none have been definitively validated or standardized. Andreasen et al (2005) suggest systematic operational criteria using eight PANSS items for which patients must score ≤ 3 (mild) for at least six months.
Using data from a one year, multi-site clinical trial (n = 675) remission criteria were compared to total PANSS scores and other endpoints and demonstrate excellent agreement with overall clinical status.
Compared to total PANSS score of 60 points and other criteria, at time points > 6 months (8 and 12 months) the specificity of the remission criteria was 85%, i.e. of the patients who had a total score >60, 85% were classified as "not in remission." Sensitivity was also very high; 75% of patients with scores of <60 were classified as "in remission."Patients who dropped out of the trial were more likely not to be in remission prior to dropping out.
These findings indicate that the remission criteria are both sensitive and specific indicators of clinical status. Additional analyses are required to determine if remission status predicts other outcomes, such as employment, independent living, and prognosis.
Prevention and recovery for patients with schizophrenia and other psychotic disorders are becoming the principle objectives for research and treatment.  While promising findings suggest that these goals will soon be attainable, efforts have also focused on development of interventions intended to reduce symptom severity over time. By maintaining low levels of psychopathology, such interventions may promote productive social and occupational pursuits and help contribute to the ultimate goal of recovery from illness. Consequently, researchers and clinicians have begun to seek consensus on the concept of symptom remission. Current methods for assessing remission vary from study to study, often utilizing measures of symptom severity developed for other purposes. In the past, remission in schizophrenia has been measured either by improvements in overall psychopathology from baseline (as measured by statistically significant percentage reductions in all thirty Positive and Negative Syndrome Scale (PANSS) items) or by reduction below specific threshold levels (e.g.: scores of less than 3, Mild in all thirty PANSS items). [2, 3]
While it makes intuitive sense to utilize instruments that are well characterized in clinical research, none have been definitively validated as measures of remission. Additionally, when used according to current standards, some interview tools (e.g. the Structured Clinical Interview for the PANSS or SCI-PANSS) require extensive time to complete as they incorporate a number of questions and probes that may not be useful for assessing remission. The process of developing and testing a measure of remission based on threshold levels of selected PANSS items should be differentiated from previous efforts to create shortened versions of existing scales (e.g. the SF-12, derived from the SF-36). Rather than substitute for the 30-item PANSS, development of a concise outcome measure for remission in schizophrenia would create a benchmark for treatment and maintenance goals in clinical research and general practice. By developing an approach based on uniform criteria rather than the various criteria currently in use, a more meaningful standard for assessing remission status can be applied in both research and clinical practice.
The Remission Construct
PANSS Items for use in Symptoms of Remission
Sources of Information
Interview & Informant Report
Unusual thought content
Interview & Informant Report
Interview & Informant Report
Passive social withdrawal
Informant Report Only
Lack of spontaneity/Flow of Conversation
Symptomatic and Functional Remission
Weiden et al. suggest that functional endpoints are not independent from psychopathology, describing symptomatic "stabilization" as a prerequisite for improvements in cognition, quality of life, and related areas.  Supporting evidence from the literature shows that specific aspects of psychopathology seem to be related to occupational function. Some reviews of the subject have critiqued psychometric rating scales for their failure to incorporate functional outcomes. This is a valid concern and one that requires additional attention and development. However, several instruments do take the impact of symptoms on function into account, such as the PANSS. While most of the PANSS items that are included in Table 1 are made largely on the basis of clinical observation, three are also based in part on functional impact over the past week. A fourth item (Passive/apathetic social withdrawal) is based solely on informant report of impairments in social function. Although these items may not provide a thorough indication of functional status, if utilized correctly, they should provide some evidence of the impact of disease severity on progress towards functional remission.
Psychometric Properties of PANSS Remission Items
Psychometric Properties of PANSS Remission Items
Sources of Information
Highest loading item for Pentagonal (0.89) and Hexagonal Positive Factor (0.81)
2nd highest loading item for Pentagonal (0.79) and Hexagonal Positive Factor (0.68)
Primary loading on Pentagonal Positive Factor (0.43) and Hexagonal Positive Factor (0.28) Secondary on Pentagonal Autistic Preoccupation Factor (0.31)
Highest loading item for Pentagonal (0.84) and Hexagonal Negative Factor (0.75)
2nd highest loading item for Pentagonal Negative Factor (0.78) and Hexagonal Positive Factor (0.65)
High loading on Pentagonal Negative Factor (0.689) and Hexagonal Withdrawal Factor (0.89)
Moderate loading on Pentagonal Negative Factor (0.44) and Hexagonal Disorganized Factor (0.28)
Highest loading on Pentagonal (0.74) and Hexagonal Disorganized Factor (0.75)
Recently, researchers have conducted retrospective analyses using these new remission criteria to determine whether remission may be used as a study outcome. Lasser et al (2005), as well as Sethuraman et al (2005) and others have used the eight PANSS items selected by Andreasen et al to identify remitted patients within randomized clinical trials and characterize symptom severity and other features. Results show that remitted patients had lower total PANSS scores, spent fewer days as inpatients, had shorter duration of illness, and used fewer prescription psychotropics.
The studies published to date have provided valuable insight into the promise of objective, standardized approaches to assessment of remission status. However, there are several outstanding questions that cannot be readily addressed without additional work. The first question is whether any consensus measure reliably identifies patients who have low levels of psychopathology. Standard statistical tests, such as sensitivity and specificity, agreement over chance, and positive predictive value must be conducted. The second question is whether the longitudinal criteria recommended by Andreasen and colleagues (i.e. maintaining low levels of psychopathology for 6 months or more) improves the validity of the measure.
The PANSS is a thirty item scale, with individual item scores from 1–7. Total scores range from 30–210. Using PANSS data from a prospective, multi-site trial, symptom severity was compared in patients in remission vs. those not in remission as defined by the operational criteria described above (scores ≤ 3 for all eight remission items). Based on work by Leucht and others, two sets of cutoffs, one for a "mildly ill" total PANSS Score (≤ 60) and a moderately low total PANSS Score (≤ 75) were used as the "gold-standards" in this analysis.  The analysis was first conducted in cross-sectional fashion at each visit, comparing total PANSS scores in subjects who meet symptom severity criteria for remission vs. non-remitted subjects.
Classification for Measures of Agreement
Low/Moderate(≤ 75), Low(≤ 60) Symptom Severity
High Symptom Severity
a. In remission, low/moderate, low symptom severity
b. In remission, high symptom severity
a + b. Total in Remission
Not In Remission
c. Not in remission, low/moderate, low symptom severity
d. Not in remission, high symptom severity
c +d. Total Not in Remission
a + c. Total Low/Moderate Low Symptom Severity
b + d. Total High Symptom Severity
Based on the cells identified in Table 3, analyses of four parameters were calculated: sensitivity, specificity, positive predictive value (PPV) and negative predictive value. Logistic regression techniques were also be used to calculate likelihood of remission status over time (with past remission status used to predict remission status at a specific time point).
The ROSE Study
This analysis is based on data collected from the ROSE Study, a one-year multi-center, open-label, randomized clinical trial carried out in 21 centers in 17 US states (Mahmoud et al, 1999). Patients were randomly assigned to treatment with either risperidone or a conventional antipsychotic drug (n = 684). Follow-up was conducted from 1995–1997. 
Inclusion criteria for the study included patients aged 18–60 y.o., diagnosed with schizophrenia or schizoaffective disorder according to ICD-9-CM criteria. For inclusion, diagnosis had to be made prior to age 35 with at least one hospitalization in a locked facility in the 2-years prior to the study, with symptom relapse at study start as defined by either (a) an inpatient admission with a mental disorder within 10 days of study start-up or (b) an emergency room visit, contact with crisis psychiatric treatment services, or (c) an unscheduled office or clinic visit accompanied by exacerbation of symptoms of schizophrenia. The PANSS was administered every 4 months, i.e. at baseline (T1), at 4 months (T2), at 8 months (T3), and at 12 months (T4).
A majority (69%) of the patients in this study were male. Average age was 38.5 y.o. (SD = 9), with 59.7% of subjects identified as White, 26.5% Black, 4.7% Hispanic, 4.6% Asian, and 4.4% in other categories. Mean age of onset of psychiatric symptoms was 22 y.o. (SD = 6), making the mean course of illness 16 years by the start of the study. The majority of subjects had never been married at the time the study was conducted (58.8%) and were inpatients (65.5%).
Total PANSS Scores for Patients by Remission Status, T2-T4
4 Month Visit (T2)
Not In Remission
8 Month Visit (T3)
Not In Remission
12 Month Visit (T4)
Not In Remission
8-Item Remission Criteria vs. Total PANSS Score < 60
8-Item Remission Criteria vs. PANSS < 75
Although positive correlations were seen when T1 remission scores were compared to T2, T3 or T4, these associations were not statistically significant. However, all associations between T2, T3 and T4 scores were significantly correlated (p < .001; r's ranged from 0.39-.4.8). When measured across all four timepoints, correlations between the Positive subscale (0.3–0.65), Negative subscale (0.41–0.66), and General Psychopathology subscale (0.30–0.60) were significantly correlated (p < 0.001).
Predicting Remission Status
Predicting Remission Status
If in Remission at Visit:
Predicted Remission Status at Visit:
95% Confidence Interval
Predicting Dropout Status
If not in Remission at Visit:
Predicted Droput Status at Visit:
95% Confidence Interval
While a consensus is developing on one definition of symptom remission, it is important to not confuse this operational definition with the broader concept of recovery. As described by Resnick and others, recovery is multidimensional; reduction of psychotic symptoms represents just one aspect of recovery. This difference is clearly articulated by in studies that use both objective clinical data, as well as subjective endpoints such as patient self-experience and measures of hopefulness. Work by Lysaker and colleagues demonstrates that the objective and subjective elements of recovery are related, but (Lysaker et al., 2006) Our analysis was intended to test the statistical properties of one operational definition, based on previously published criteria.
Based on our findings, the remission criteria of Andreasen et al appear to be potentially valuable for future studies. First, these criteria demonstrate high specificity, sensitivity, and positive predictive value when compared to PANSS total score. The results of cross-sectional comparisons of two cut-offs of the total PANSS score (60 and 75) clearly show almost all subjects identified as being in remission at a given time point have low/low-moderate levels of psychopathology. At first glance, the results of this analysis at T1 (baseline visit) suggest that the remission criteria are not valid. However, it should be noted that very few subjects met the remission criteria at baseline (n = 30), making it difficult to draw any meaningful conclusions regarding total PANSS scores. By contrast, when study-defined relapse is examined, the remission criteria appear to have excellent predictive value.
Next, when longitudinal components of the criteria are included, the predictive value of the criteria improves. Specifically, using data from T2-T4, for subjects in remission at 4 months and 8 months, the likelihood of being in remission at 12 months is increased 10-fold versus those subjects who fail to be in remission at one or both of the preceding time points. The analysis of data from T1-T3 is problematic since exceptionally few subjects met criteria at T1, as noted previously. This may account in part for the trend towards significance and the wide confidence interval. The differences in the value of the remission criteria at T3 to T4 are only observable when available data from previous visits are included.
An interesting, if unexpected finding was the strong relationship between failure to achieve remission and risk of dropping out of the study. Similar results have been reported previously; Hatta et al. suggest that "treatment resistance" may be predicted using PANSS baseline scores on selected items, while other studies relate medication non-compliance to PANSS-measured mood symptoms and PANSS total score. [11–13] Through quantitative approaches to treatment (i.e. "targeted treatment" – see Opler et al., 2006 for a review) such findings are gradually being applied in various clinical settings. In future research, efforts might be made to study the reasons for medication discontinuation and correlations with psychopathology. 
One of the major criticisms of analyses based on samples drawn from randomized clinical trials is that they may not resemble the general population. Certain types of subjects are potentially excluded or under-represented (e.g. the elderly, those with co-morbid conditions, such as drug dependence).  In order to test the validity of the remission construct, particularly the extent to which it is generalizable to all patients with schizophrenia spectrum disorders, samples drawn from multiple populations should be compared.
The primary limitation of this analysis is that it does not incorporate an independent measure of functional status. While there is some data showing that changes in PANSS scores are related to quality of life measures, determining the best approach towards quantifying remission will require that both psychopathology and function be taken into account.  Although not generally acknowledged, the PANSS and similar scales are intended to incorporate both observed and reported impact of symptoms on function. In patients suffering from acute psychosis, the lack of insight necessitates the use of 'informants', or third-parties who can contribute data on the patient's behavior and activities over the past week. (Note: The clinician performing the interview is permitted to act as the informant if he or she has specific knowledge about the patient over the past week.)  While the conventions of the PANSS require that informant data be obtained and incorporated, it is rarely considered separately from the scale itself. In future research, informant data might be analyzed separately and jointly in the interest of determining whether a meaningful "functional impact factor" could be extracted from PANSS data, compared to independent, previously validated measures and potentially applied to new studies of remission in schizophrenia.
While the eight items included in this construct are positively associated with selected outcomes and have several useful properties, it is necessary to take an empirical approach to this question as well, i.e. utilizing statistical techniques to objectively determine which PANSS items are most strongly associated with remission over time. One possibility is the application of Item Response Theory to determine which PANSS items are most valuable for differentiating remitted and non-remitted patients; such approaches have been taken with depression rating scales. 
A practical concern is the need for standardized, efficient methods for data collection.
In conclusion, our findings suggest that the operational criteria outlined by Andreasen and colleagues have potential for use in the future. However, we also conclude that our results are limited in several respects and key endpoints must be incorporated in future studies. Going forward, we intend to compare multiple operational criteria objectively, using rigorous techniques. In doing so, symptom remission should be considered as one aspect of the broader construct of recovery from illness – while future studies may incorporate symptom remission as one aspect of recovery (or possibly as one milestone in the process of recovery) the field must continue to strive for the best possible outcomes for our patients. Given the urgent need to continue to improve on "real-world" efficacy of antipsychotic treatment , no construct should be considered sacred and expectations should not be lowered based on limited data.
The authors wish to acknowledge Drs. Leonard White and Lewis A. Opler for their valuable guidance and intellectual support.
- Corrigan PW: Recovery from schizophrenia and the role of evidence-based psychosocial interventions. Expert Rev Neurother. 2006, 6 (7): 993-1004. 10.1586/1473722.214.171.1243.View ArticlePubMedGoogle Scholar
- Sethuraman G, Taylor CC, Enerson M, Dunayevich E: A retrospective comparison of cumulative time spent in remission during treatment with olanzapine or risperidone among patients with schizophrenia. Schizophr Res. 79 (2–3): 337-40. 2005 Nov 15
- Kissling W, Heres S, Lloyd K, Sacchetti E, Bouhours P, Medori R, Llorca PM: Direct transition to long-acting risperidone–analysis of long-term efficacy. J Psychopharmacol. 2005, 19 (5 Suppl): 15-21. 10.1177/0269881105056514.View ArticlePubMedGoogle Scholar
- van Os J, Drukker M, A Campo J, et al: Validation of remission criteria for schizophrenia. Am J Psychiatry. 2006, 163: 2000-02. 10.1176/appi.ajp.163.11.2000.View ArticlePubMedGoogle Scholar
- Docherty JP, Bossie CA, Lachaux B, Bouhours P, Zhu Y, Lasser R, Gharabawi GM: Patient-based and clinician-based support for the remission criteria in schizophrenia. Int Clin Psychopharmacol. 2007, 22 (1): 51-5.PubMedGoogle Scholar
- Weiden P, Zygmunt A: Medication noncompliance in schizophrenia. Part I. Assessment. J PracPsychiatry Behav Health. 1997, 3: 106-110.Google Scholar
- White L, Harvey PD, Opler L, Lindenmayer JP: Empirical assessment of the factorial structure of clinical symptoms in schizophrenia. A multisite, multimodel evaluation of the factorial structure of the Positive and Negative Syndrome Scale. The PANSS Study Group. Psychopathology. 1997, 30 (5): 263-274.View ArticleGoogle Scholar
- van den Oord EJ, Rujescu D, Robles JR, Giegling I, Birrell C, Bukszar J, Murrelle L, Moller HJ, Middleton L, Muglia P: Factor structure and external validity of the PANSS revisited. Schizophr Res. 2005,Oct 14.
- Lasser RA, Bossie CA, Gharabawi GM, Kane JM: Remission inschizophrenia: Results from a 1-year study of long-acting risperidoneinjection. Schizophr Res. 77 (2–3): 215-27. 2005 Sep 15
- Leucht S, Kane JM, Kissling W, Hamann J, Etschel E, Engel RR: What does the PANSS mean?. Schizophr Res. 79 (2–3): 231-8. 2005 Nov 15
- Mahmoud R, Engelhart L, Ollendorf D, Oster G: The Risperidone Outcomes Study of Effectiveness (ROSE):A Model for Evaluating Treatment Strategies in Typical Psychiatric Practice. J Clin Psychiatry. 1999, 60 (suppl 3): 42-47.PubMedGoogle Scholar
- Hatta K, Nakamura H, Matsuzaki I: Acute-phase treatment in general hospitals: clinical psychopharmacologic evaluation in first-episode schizophrenia patients. Gen Hosp Psychiatry. 2003, 25 (1): 39-45. 10.1016/S0163-8343(02)00252-9.View ArticlePubMedGoogle Scholar
- Kozuki Y, Schepp KG: Adherence and nonadherence to antipsychotic medications. Issues Ment Health Nurs. 2005, 26 (4): 379-96. 10.1080/01612840590922425.View ArticlePubMedGoogle Scholar
- Hudson TJ, Owen RR, Thrush CR, Han X, Pyne JM, Thapa P, Sullivan G: A pilot study of barriers to medication adherence in schizophrenia. J Clin Psychiatry. 2004, 65 (2): 211-6.View ArticlePubMedGoogle Scholar
- Opler L, Malaspina D, Opler M: Reducing guesswork in schizophrenia treatment. Current Psychiatry. 2006, 5 (9): 76-84.Google Scholar
- Rabinowitz J, Bromet EJ, Davidson M: Are patients enrolled in first episode psychosis drug trials representative of patientstreated in routine clinical practice?. Schizophr Res. 61 (2–3): 149-55. 2003 Jun 1
- Phillips GA, Van Brunt DL, Roychowdhury SM, Xu W, Naber D: The relationship between quality of life and clinical efficacy from a randomized trial comparing olanzapine and ziprasidone. J Clin Psychiatry. 2006, 67 (9): 1397-403.View ArticlePubMedGoogle Scholar
- The PANSS Manual. Kay, Opler, and Fiszbein: ©s. 2006, Multi-Health Systems, Inc., Toronto, CA
- Gorwood P: Meeting everyday challenges: antipsychotic therapy in the real world. Eur Neuropsychopharmacol. 2006, 16 (Suppl 3): S156-62. 10.1016/j.euroneuro.2006.06.002. Epub 2006 Jul 25View ArticlePubMedGoogle Scholar
- Lysaker PH, Buck KD, Hammoud K, Taylor AC, Roe D: Associations of symptoms, psychosocial function and hope with qualities of self-experience in schizophrenia: comparisons of objective and subjective indicators of health. Schizophr Res. 2006, 82: 2-3. 10.1016/j.schres.2005.12.844.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-244X/7/35/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.