Results of the present study showed that, within the subgroup of patients with an unmet need for care with respect to psychotic symptoms at baseline, patients receiving FACT were more likely to be in remission than were patients receiving standard treatment. This suggests that FACT makes a difference in patients for whom either the patient or the clinician recognises an unmet need in this area. However, a coexisting alcohol problem precluded any associations with FACT.
A first explanation for only finding an association between FACT and remission in patients scoring an unmet need on psychotic symptoms could be that these patients may be the most severely ill. The conclusion that FACT is more effective than standard care for the high care patient subgroup and should be restricted to these patients, could be a consequence of a "regression to the mean" effect. However, if severity was the only explanation, we would have expected an interaction between severity of symptoms and FACT. Post hoc, an interaction term between FACT and BPRS positive symptoms was added to the regression model. This interaction term was statistically imprecise by conventional alpha, while the interaction between FACT and CAN psychotic symptoms remained. In addition, patients with an unmet need in psychotic symptoms are not per definition the most severely ill [see [29]], although it is likely that severely ill psychotic patients will have a need for care on that item. This is verified by the correlation between unmet need with respect to psychotic symptoms and BPRS positive symptoms, which is relatively high, but not perfect (r2 = 0.6) and higher than the other three BPRS sum scores (r2 between 0.17 and 0.21, all correlations p < 0.001).
There are several other explanations for finding an association in patients with an unmet need with respect to psychotic symptoms, only. First, in standard care, a patient with a need for care with respect to psychotic symptoms is treated by a single professional, often a psychiatrist who provides medication to manage symptom exacerbation. In FACT, a multidisciplinary team provides treatment for all SMI-patients rather than only for those SMI patients who need intensive care. This team includes a psychiatrist, psychologists, addiction workers, case managers and vocational rehabilitation workers using Individual Placement and Support (IPS). Professionals from the multidisciplinary team start treatment in their area of expertise, in parallel to the psychiatrist who reduces psychotic symptoms. Because of the continuity-of-care principle these interventions are continued when crises are managed.
Second, over the years, some ACT related principles became 'good practice' references and were included in standard care. This could be a reason why no extra beneficial effects could be shown in the group of patients without a need for care with respect to psychotic symptoms.
The present study also showed that a need for care in the area of alcohol precluded the beneficial effects of FACT. Patients with alcohol dependence do not seem to benefit from FACT. This is not in line with a previous study showing that high fidelity to ACT (including dual disorder treatment) is of particular importance for substance abuse outcomes in dual disorder patients ([39]. In Maastricht the fidelity to ACT guidelines is acceptable (see below), but addiction 'specialists' were not yet available during the study assessment period. SMI patients with comorbid alcohol addition may be more difficult to treat than other SMI patients, resulting in increased use of crisis services and higher percentages of self-harm [24]. In a London study, dual disorder patients were more often hospitalised or involuntary admitted and the authors speculated that these patients might benefit from specific interventions, such as a specialized FACT team for dual disorder patients [25]. Fidelity guidelines for FACT-teams in the Netherlands do list Integrated Dual Diagnosis Treatment as a core feature and FACT-teams in other Dutch regions did integrate these. Currently Integrated Dual Diagnosis Treatment is progressively implemented in the Maastricht FACT teams. It is expected that this would improve outcomes for dual disorder patients.
Other European and recent US studies showed less impressive effect sizes than the present study [12, 17, 19–21]. The focus of FACT on early detection of symptom exacerbation to prevent more serious symptomatology, may explain the difference [7]. Another reason could be fidelity, because lower fidelity has been associated with a lower effectiveness [6, 14, 39]. Preliminary results using the DACTS – a scale measuring ACT fidelity [40] showed that two Maastricht FACT teams had acceptable scores on fidelity for ACT, while a third team had a moderate score (personal communication Van Vugt). A post-hoc analysis restricted to patients of the two teams with acceptable fidelity and their matched controls showed associations between FACT and symptomatic remission that were similar to the original analysis (in patients with an unmet need on psychotic symptoms: OR = 7.06; p = 0.004; 95% confidence interval = 1.8 – 27.1, n = 88; in patients with both an unmet need on psychotic symptoms and no need with respect to alcohol: OR = 8.39, p = 0.003, 95% CI = 2.11 – 33.4, n = 67). Recently, a Dutch expert group embedded in the Centre for Certification ACT & FACT http://www.ccaf.nl evaluated working ingredients and formulated fidelity criteria to evaluate FACT rather than ACT. This resulted in the development of a FACT Fidelity Scale (FACTs) and currently field tests are being conducted.
As a result of the integrated patient mix in FACT, it was impossible to isolate the cases within the FACT-teams that were effectively receiving ACT in the present study. Therefore, this is one of the first FACT outcome studies. Further research is needed to replicate the present results. These future studies should include interaction terms and they should assess and control for the fidelity of the FACT teams. It is also important that future studies clearly describe the ingredients of FACT as well as standard treatment, so that effective elements can be identified.
In sum, results of the present study support the recommendation of the Dutch schizophrenia guidelines, which recommend ACT as the primary treatment for people with SMI [1]. Within the Dutch mental health care system this can be operationalised by starting FACT teams. The addition of an addiction specialist to the teams may improve benefits for patients with a coexisting alcohol problem. Although only patients with an unmet need on psychotic symptoms seemed to benefit from FACT, we feel that all patients with or without an unmet need in this area should remain in FACT, because continuity of care is an important feature, probably contributory to the effectiveness of FACT.
Methodological issues
The strength of the present study is the unique data collection, as real-life observational data were obtained longitudinally within the treatment process and interviews were conducted by different interviewers in different settings. This might increase the generalizability of the results. However, it can also threaten reliability. To minimize this threat, a manual for the assessment of the interviews was developed, interviewers were trained on a regular basis (booster sessions) and new mental health care professionals received supervision in the scoring of CNCR interviews. Another possible threat to the reliability of the data collection is that the interviewers were the mental health care workers involved in the treatment of the subjects and that they were not blinded for the treatment condition. This is, however, hypothesized not to pose a big threat to reliability as the assessment of CNCR interviews were in place years before the introduction of the FACT teams and were never positioned in an evaluation of FACT.
A second important strength of the present study is its outcome measure. Symptomatic remission has been shown to be a clinically meaningful concept and is a feasible outcome measure [36]. The outcome measures most commonly used in studies of the effectiveness of ACT/FACT are the number and/or length of hospital admissions ([10, 17, 19, 27, 41], but several studies have shown that these outcomes correlate more with the number of available beds than with the service that is being delivered [27, 41]. Consequently, the present study used a clinical outcome measure. However, the definition of full remission, rather than symptomatic remission, is not only based on 7 BPRS items, but also includes a time criterion: symptom levels should remain low for at least 6 months. This time criterion could not be incorporated because only a few patients in this sub sample were measured more than twice and observations were mostly more than 1 year apart. Bak and colleagues showed in a sensitivity analysis that when remission was defined as scoring low on the 7 BPRS items at two successive moments, results were even stronger than the original results [18]. Therefore, it is likely that including the time criterion would have shown similar or even stronger results in favour of FACT, rather than weaker effect sizes.
In addition, this is a naturalistic study of regular clinical practice and the CNCR started in 2004 in the control region, while the first FACT team was established in 2002. Therefore, data from FACT patients and their matched controls could be a maximum of two years apart and the first included assessment occurred after FACT introduction. Furthermore, some best-match controls contributed only one assessment. Analyses may have been methodologically more sound if changes since baseline were analysed, by controlling for remission at baseline and excluding the baseline assessment. Two other ACT studies studied changes in symptom severity [12, 17] and these studies did not show any effect. However, according to Andreasen and colleagues [35] 'the real-world interpretability of change scores as a primary outcome is limited because of the variability of baseline symptom severity across intervention trials'.
The current data differ from those used by Bak and colleagues [18] because information from the PCR as well as the CNCR was used to identify which patient actually received FACT. However, the data of the merged PCR and CNCR databases did not always concur with respect to service delivery: some patients were identified as being a FACT patient in one database but not in the other. Therefore, a person was assigned to the FACT group if he/she was classified as receiving FACT at least once in either of the databases. One reason for these differences is that PCR data from the second half of 2006 and from 2007 were not available yet. Although it is possible that some FACT patients were missed, none of these FACT patients were included in the control group, because these were obtained from an adjacent region.
Propensity score matching is a relatively new method of matching which can be used in observational data where treated subjects differ systematically from controls [42]. However, this type of matching can be performed in several different ways and it is unclear which option is the best. Therefore, we performed nearest neighbour matching both with and without replacement [43]. Fortunately, results of the different analyses were rather similar. In addition, an extra level was included in the multilevel analyses to control for the matching [42].
The present study has some other limitations. First, non-FACT patients originated from an adjacent region in which FACT had not yet been introduced. However, the South Limburg patient population is similar in the two sub regions and stable over time. Therefore, invalidation of the results is negligible.
Second, matching, using the first measurement after the start of FACT could have obscured the effect of FACT (FACT patients and controls are similar because the patients outcomes already improved). The current analysis strategy is the best possible to obtain ecologically valid results and the reported association between FACT and remission cannot be a result of the matching using first measurement data.
Third, the CNCR protocol prescribes that CNCR interviews should be carried out at intake, as part of the yearly evaluation, and at every mutation in the patient's care plan. However, due to pragmatic and logistic reasons, professional carers do not always comply exactly. It is possible that members of FACT-teams are better compliers than professional carers in standard treatment, because the CNCR interview is implemented within FACT. In addition, FACT aims to serve patients throughout their course of illness, also offering low-profile supportive care while standard care patients with the same severity of the symptoms are transferred to a less intensive type of care. Therefore, patients in remission may be interviewed more often when served by a FACT-team and this could explain part of the effect reported in the present paper. However, reminders are routinely sent to all interviewers who do not turn in the yearly reassessment and by matching CNCR data with PCR data it is in future also possible to send reminders with every mutation in treatment. Unfortunately, this feedback procedure has a delay of some months. Furthermore, the management in the control region enforces the professional carers to fill in the CNCR forms. Finally, the helpdesk in the control region is very strict in sending reminders and professional carers are also personally approached if necessary. Therefore, we feel that the difference in compliance cannot be that large to fully explain the positive findings.
Furthermore, reliability of the GAF at the individual level is not sufficient and this could lead to random misclassification in the propensity scores [44]. Some FACT patients may, therefore, not have been matched to the "closest" control patient. However, because this misclassification is random it only leads to noise and a larger confidence interval, while the effect size remains.
Despite the limitations, it is a unique study that combines the merits of an observational study with a careful statistical procedure, resulting in promising results for FACT in the Netherlands.
