A real-life observational study of the effectiveness of FACT in a Dutch mental health region

PCR and CNCR

The data for the present study were obtained from a merged database consisting of the Psychiatric Case Register (PCR) and the Cumulative Needs for Care Register (CNCR). The PCR registers the consumption of mental health care of psychiatric patients in South Limburg (population 660 000). Data for the PCR have been provided by all regional mental health facilities for all patients since 1981 [28]. The CNCR monitors the need for care of patients with SMI in South Limburg. The data for the CNCR have been collected in Maastricht since 1998 and in the other regions of South Limburg since 2004. Data are cumulatively and routinely collected in clinical practice with intervals of 1 to 2 years and with every major change in treatment or setting. There is no specific baseline in the flow of the illness history and the number of follow-up measurements differs per person as this is a naturalistic study [29]. For the present analyses, the CNCR and PCR were anonymously matched using a case identification code with an encryption algorithm and the date of the CNCR interview [18] and values of the first assessment after 2002 were used as a proxy for baseline values.

CNCR interview

A CNCR interview [29] provides information on (i) demographic variables, (ii) current level of psychopathology, (iii) level of general functioning, (iv) needs for care, v) quality of life (5 items) and vi) quality of care (1 item).

Psychopathology was assessed using the Brief Psychiatric Rating Scale (BPRS). The BPRS [30] is a semi-structured interview assessing the presence and severity of various symptoms in the two weeks preceding the interview. The BPRS items are scored on a 7-point Likert scale (1 = not present; 7 = most severe). Ratings are based upon clinical observations of symptoms and patients' verbal report of symptoms. In order to standardize the ratings, anchor points and probe questions are described for each item [29, 30]. Factor analyses of the BPRS have revealed the existence of four underlying constructs ('BPRS scales'): negative symptoms, positive symptoms, manic excitement and depression/anxiety [31]. The BPRS items blunted affect, motor retardation, emotional withdrawal and self-neglect loaded on negative symptoms; bizarre behaviour, unusual thought content, disorientation, hallucinations and suspiciousness loaded on positive symptoms; motor hyperactivity, elevated mood, excitement, distractibility, hostility and grandiosity loaded on manic excitement; and depression, anxiety, suicidal ideation and guilt loaded on depression/anxiety.

Need for care was assessed using the Camberwell Assessment of Need (CAN) [32]. Baseline values of a few individual items were used in the present paper (see statistical analysis) because the sum score lacks validity and, therefore, is likely of less value [33]. The CAN includes 22 items (e.g. daytime activities, psychotic symptoms). All CAN items can be scored 0 (no problem), 1 (there was a problem, but the problem is met), 2 (unmet need) with a reference period including the last 3 months [8]. In the CNCR, information from clinician and patient is combined using a priori decision rules to maximise the clinical relevance of the rating.

Information on the global functioning of patients was obtained with the extended Global Assessment of Functioning (GAF) scale [34]. This tool consists of a psychopathology scale (GAF-p, sample range 1–95), rating the global symptom severity of patients, and an impairment scale (GAF-i range 1–95), rating their level of impairment in psychosocial functioning.

Outcome measure

The primary outcome measure was symptomatic remission of psychotic disorder. Symptomatic remission is defined as a symptom intensity level within the normal general population range, reflecting negligible influence on an individual's functioning [35]. For the objective measurement of symptomatic remission the following 7 BPRS items are included [35, 36]: 1) grandiosity, 2) suspiciousness, 3) unusual thought content, 4) hallucinatory behaviour, 5) conceptual disorganization, 6) mannerism/posturing and 7) blunted affect [35, 36]. These BPRS items are diagnostically specific to psychotic disorders. They reflect both the three dimensions of psychopathology underlying psychotic disorders (psychoticism, disorganization, negative symptoms) and the five acute phase criteria of schizophrenia in the DSM IV (delusions, hallucinations, disorganization, disorganized or catatonic behaviour, negative symptoms) [35, 36]. In order to be classified as being in symptomatic remission, patients had to score 3 or less on all 7 BPRS items [35, 36].

Subjects and matching

Data of SMI patients collected between April 2002 (the start of FACT in the Maastricht region) and August 2007 were included in the matching procedure. Of the matched patients, data up to December 2007 were included in the multilevel analyses (see below). Patients were classified as SMI patients if they fit the diagnosis of schizophrenia or psychotic disorder (DSM-IV 295, 297 or 298), or had a minimal score of 15 on the positive symptom scale of the BPRS on any of the assessments. Additionally, patients with a low level of functioning (below 45 on at least one of the GAF scales) and a minimum of 2 needs in the domains of accommodation, welfare benefits, alcohol and drugs, were classified as SMI patients [33]. These four domains of needs were a priori selected from the 22 CAN domains. Because services for alcohol and drug addiction did not take part in the data collection, patients with substance abuse disorder with no further psychiatric disorders were not included. As the definition was to a degree arbitrary and an absolute cut-off may be too strict, the group of patients scoring below 45 on one of the two GAF scales and a need in 1 of the 4 above-mentioned CAN domains was also included in the analyses as moderately mentally ill (MMI) rather than SMI patients. All patients not meeting the above criteria for MMI and SMI were excluded from the analyses, even if they were treated by a FACT-team.

Next, all patients ever identified by either the PCR or the CNCR as receiving FACT were coded as FACT patients. FACT patients are a more severely ill subgroup of all patients in the CNCR database. Consequently, patients coded as receiving FACT were matched with controls, using propensity score nearest neighbour-matching (one on one) with replacement. Propensity scores were based on age category (18–30, 31–65, 66+), the two GAF-scales and the four BPRS sum scores (all continuous variables). The FACT group included Maastricht patients only, while Maastricht patients were excluded when selecting matched controls, because all Maastricht patients who were eligible for FACT were already served by one of the FACT teams. Before matching, FACT patients differed significantly from all non-FACT patients with respect to age (mean age FACT-patients 38 years, others 43 years) and GAF (mean symptoms FACT-patients 52.9, others 50.7). Positive symptoms and negative symptoms were higher in the FACT group and depression/anxiety was lower, but differences were not statistically significant. After the matching, matching variables were more balanced than before, although there was still a small difference in age (mean 37 years vs 39 years, p = 0.08).

Statistical analysis

All analyses were performed using the statistical program Stata 10 [37]. Since the data consisted of multiple measurements per patient and the patients were matched the observations were not independent. As a result, standard logistic regression techniques could not be used for statistical analysis of the data. The data with measurements (level 1) clustered in subjects (level 2) clustered in matched groups (level 3) were, therefore, subjected to multilevel logistic regression analysis, which is ideally suited for analysis of this type of data [38]. The xtmelogit command is the logistic regression variant of the Stata xtmixed command with 2 or more levels. The group level was added to control for clustering in the data introduced by the matching. The dependent variable was symptomatic remission (yes/no) a variable at measurement level, while the independent variable was FACT (yes/no). Age, symptom severity at baseline (four BPRS continuous variables), functioning at baseline (two GAF scales, continuous) and gender were included in the models as independent variables.

In addition, interaction terms between FACT and a set of baseline variables as outlined earlier were included in the model: GAF symptoms and handicap, level of chronicity, CAN alcohol, CAN drugs, CAN psychotic symptoms and CAN self care. When any of these interaction terms was statistically significant (alpha was set at 0.05) the Stata Lincom procedure was used to calculate odds ratios of FACT and remission for all strata of the interaction variables.

Methodological issues

The strength of the present study is the unique data collection, as real-life observational data were obtained longitudinally within the treatment process and interviews were conducted by different interviewers in different settings. This might increase the generalizability of the results. However, it can also threaten reliability. To minimize this threat, a manual for the assessment of the interviews was developed, interviewers were trained on a regular basis (booster sessions) and new mental health care professionals received supervision in the scoring of CNCR interviews. Another possible threat to the reliability of the data collection is that the interviewers were the mental health care workers involved in the treatment of the subjects and that they were not blinded for the treatment condition. This is, however, hypothesized not to pose a big threat to reliability as the assessment of CNCR interviews were in place years before the introduction of the FACT teams and were never positioned in an evaluation of FACT.

A second important strength of the present study is its outcome measure. Symptomatic remission has been shown to be a clinically meaningful concept and is a feasible outcome measure [36]. The outcome measures most commonly used in studies of the effectiveness of ACT/FACT are the number and/or length of hospital admissions ([10, 17, 19, 27, 41], but several studies have shown that these outcomes correlate more with the number of available beds than with the service that is being delivered [27, 41]. Consequently, the present study used a clinical outcome measure. However, the definition of full remission, rather than symptomatic remission, is not only based on 7 BPRS items, but also includes a time criterion: symptom levels should remain low for at least 6 months. This time criterion could not be incorporated because only a few patients in this sub sample were measured more than twice and observations were mostly more than 1 year apart. Bak and colleagues showed in a sensitivity analysis that when remission was defined as scoring low on the 7 BPRS items at two successive moments, results were even stronger than the original results [18]. Therefore, it is likely that including the time criterion would have shown similar or even stronger results in favour of FACT, rather than weaker effect sizes.

In addition, this is a naturalistic study of regular clinical practice and the CNCR started in 2004 in the control region, while the first FACT team was established in 2002. Therefore, data from FACT patients and their matched controls could be a maximum of two years apart and the first included assessment occurred after FACT introduction. Furthermore, some best-match controls contributed only one assessment. Analyses may have been methodologically more sound if changes since baseline were analysed, by controlling for remission at baseline and excluding the baseline assessment. Two other ACT studies studied changes in symptom severity [12, 17] and these studies did not show any effect. However, according to Andreasen and colleagues [35] 'the real-world interpretability of change scores as a primary outcome is limited because of the variability of baseline symptom severity across intervention trials'.

The current data differ from those used by Bak and colleagues [18] because information from the PCR as well as the CNCR was used to identify which patient actually received FACT. However, the data of the merged PCR and CNCR databases did not always concur with respect to service delivery: some patients were identified as being a FACT patient in one database but not in the other. Therefore, a person was assigned to the FACT group if he/she was classified as receiving FACT at least once in either of the databases. One reason for these differences is that PCR data from the second half of 2006 and from 2007 were not available yet. Although it is possible that some FACT patients were missed, none of these FACT patients were included in the control group, because these were obtained from an adjacent region.

Propensity score matching is a relatively new method of matching which can be used in observational data where treated subjects differ systematically from controls [42]. However, this type of matching can be performed in several different ways and it is unclear which option is the best. Therefore, we performed nearest neighbour matching both with and without replacement [43]. Fortunately, results of the different analyses were rather similar. In addition, an extra level was included in the multilevel analyses to control for the matching [42].

The present study has some other limitations. First, non-FACT patients originated from an adjacent region in which FACT had not yet been introduced. However, the South Limburg patient population is similar in the two sub regions and stable over time. Therefore, invalidation of the results is negligible.

Second, matching, using the first measurement after the start of FACT could have obscured the effect of FACT (FACT patients and controls are similar because the patients outcomes already improved). The current analysis strategy is the best possible to obtain ecologically valid results and the reported association between FACT and remission cannot be a result of the matching using first measurement data.

Third, the CNCR protocol prescribes that CNCR interviews should be carried out at intake, as part of the yearly evaluation, and at every mutation in the patient's care plan. However, due to pragmatic and logistic reasons, professional carers do not always comply exactly. It is possible that members of FACT-teams are better compliers than professional carers in standard treatment, because the CNCR interview is implemented within FACT. In addition, FACT aims to serve patients throughout their course of illness, also offering low-profile supportive care while standard care patients with the same severity of the symptoms are transferred to a less intensive type of care. Therefore, patients in remission may be interviewed more often when served by a FACT-team and this could explain part of the effect reported in the present paper. However, reminders are routinely sent to all interviewers who do not turn in the yearly reassessment and by matching CNCR data with PCR data it is in future also possible to send reminders with every mutation in treatment. Unfortunately, this feedback procedure has a delay of some months. Furthermore, the management in the control region enforces the professional carers to fill in the CNCR forms. Finally, the helpdesk in the control region is very strict in sending reminders and professional carers are also personally approached if necessary. Therefore, we feel that the difference in compliance cannot be that large to fully explain the positive findings.

Furthermore, reliability of the GAF at the individual level is not sufficient and this could lead to random misclassification in the propensity scores [44]. Some FACT patients may, therefore, not have been matched to the "closest" control patient. However, because this misclassification is random it only leads to noise and a larger confidence interval, while the effect size remains.

Despite the limitations, it is a unique study that combines the merits of an observational study with a careful statistical procedure, resulting in promising results for FACT in the Netherlands.