Measurement of outcomes

This study did not involve research interviews with patient participants. All data were gathered from case notes and interviews with unit staff. This approach was pragmatic given the fact that the two researchers (NG and IH) were concurrently recruiting and interviewing patients in a separate phase of the REAL study and it minimised the burden of research interviews on patients. Participant data were gathered at recruitment by the researchers from review of case notes as follows: demographics (age, gender, ethnic group); diagnosis; length of history; length of current admission; previous admissions; previous involuntary admissions; risk history. The researchers carried out face to face interviews with a member of the unit staff who knew each participant well (such as their primary nurse) to complete the following standardised measures: social function was assessed using the Life Skills Profile [12] which comprises 39 items, each rated on a four point likert scale with the most positive response scoring 4 and the least scoring 1, giving an overall score ranging from 39 to 156; engagement in activities was assessed using the staff rated version of the Time Use Diary [13] which assesses patients’ activities for the previous week during four periods each day - morning, lunchtime, afternoon and evening. The level of engagement in activity, and its complexity, is rated on a scale of 0 to 4 for each time period, giving a maximum possible score of 112 with higher scores denoting greater activity; patients’ function was assessed using the Global Assessment of Functioning [14] which provides an overall rating from 1 to 100 with higher scores denoting better functioning. Potential mediators of outcomes were also assessed using standardised measures that staff completed in face to face interviews with the researchers. Patients’ use of substances was assessed using the Clinician Alcohol and Drug Use Scales [15] and any challenging behaviours were assessed using the Special Problems Rating Scale [16]. Quality of care in the unit was rated by the unit manager using the QuIRC [10, 11] (staffing of the unit and availability of specific interventions are included in the QuIRC).

At 12 month follow-up, the researchers contacted a key informant to clarify whether the patient had been successfully discharged and if so, details of community placements were recorded. Where the patient had been discharged from the inpatient unit, the key informant was either the patient’s community care co-ordinator or their keyworker at their supported accommodation and for those who remained in the inpatient unit at follow-up, it was their primary nurse. Key informants were also asked to complete the Life Skills Profile [12] to re-assess social functioning. Data on length of admission in the rehabilitation unit were available from the unit manager and case notes and therefore available for all patients recruited to the study.

Measurement of service use and costs

We used an adapted version of the Client Service Receipt Inventory [17] to collect information on the number of contacts with health care professionals in the unit and in the community over the last month at both time points (baseline and 12 month follow-up). We used data compiled by Curtis [18] to cost these service contacts assuming that each had an average duration of 30 min unless information from a comparable study was available to suggest otherwise [19].

Data management

Data were entered into the study’s Access database by the researchers. Range and logic checks were built in to assist with data cleaning. Ten percent of data were double entered to check for data entry errors with an error rate set at 5 %, above which all data would be double entered. The error rate was less than 5 % and thus no double data entry was required. Data were further checked and cleaned by the statistician before analysis.

Data analysis

Data were analysed using Stata version 12 [20]. Descriptive characteristics (demography and clinical and service characteristics) were summarised using mean (SD), median (interquartile ranges) or proportions as appropriate at baseline and 12 month follow-up. Summary statistics were calculated for all standardised assessments at baseline and follow up, providing percentages or mean (SD), median (interquartile ranges) as appropriate.

Random effects regression models were used to investigate the relationship between patient characteristics, service factors (QuIRC domain scores) and outcomes. This type of regression modelling takes account of clustering of patients within units [21]. Length of stay was log transformed due to violation of normality assumptions as assessed by residual plots. When fitting the models we followed the rule of 10 events per variable [22] to ensure that the coefficients were estimated with adequate precision. Results are reported as estimates with their 95 % confidence intervals. Complete case analysis was performed, meaning that only service user participant data and unit data were included in the analsyis where there were no missing values [23]. The results are treated as exploratory since no specific hypotheses were being tested.

A further intuitive analysis was carried out to identify service and patient factors that were associated with successful discharge and a composite outcome combining those successfully discharged with those considered by staff to be ready for discharge but awaiting a suitable community placement. Categorical or binary explanatory variables with less than 5 % prevalence were omitted from this analysis to avoid model fitting problems. Univariable analysis of prespecified factors considered to be potentially associated with the outcome of interest was first carried out. Variables with a p-value of less than or equal to 0.15 were considered in the mutivariable model. Backwards elimination was then carried out with a significance level of 5 %.

We compared baseline and follow-up cost of contacts and estimated the statistical significance of the change in total costs over time using a non-parametric bootstrap approach because of the skewed cost distribution. In addition, we estimated the association between baseline service user characteristics and baseline and follow-up cost of service contacts using a random effects regression model with robust standard errors, with latter used to also address the skewed data.

Main findings

Over half the patients achieved a successful community discharge within the 12 month follow-up period and there were small improvements in the ratings of social function. The multivariable regression models did not appear to identify any associations between the seven QuIRC domains assessing the quality of care provided in the units and better clinical outcomes for patients (social function or length of admission in the rehabiliation unit). However, a number of potential factors that were associated with successful discharge were identified, (albeit with relatively small odds ratios). This analysis was repeated including patients who were considered by staff to be ready for discharge but who were awaiting a suitable community placement, in order to avoid bias due to lack of availability of appropriate local supported accommodation. Both analyses found that the degree to which patients were engaged in activities at recruitment was positively associated with successful discharge/readiness for discharge but those who had been in hospital longer were less likely to achieve this positive outcome. In the first model, patients’ social skills at recruitment (the communication sub-scale of the Life Skills Profile) were also found to be positively associated with successful discharge, as was the degree to which the rehabiliation unit operated with a “recovery” orientation.

Although the negative association between fire setting and discharge/readiness for discharge is not surprising, less easy to explain is the negative association between receiving CBT and successful discharge. These findings are discussed further below.

The cost analysis showed that there was a decrease in costs of care over the 12 months of the cohort study, though the cost of contacts with support workers remained more or less stable and the costs of contacts with care co-ordinators increased. The majority of staff in supported accommodation that most service users moved on to were support workers, explaining the stability of costs associated with this staff group. Quality of care was not associated with costs of care when adjusted for service user age, gender and social functioning. Less severe symptoms and higher functioning (higher scores on the Global Assessment of Functioning) were associated with lower costs of care, presumably because service users who were less unwell had less need of staff support. This is something that has been demonstrated previously [25]. Service users with a history of fire setting were less likely to be discharged in our sample. Therefore, there was a trend towards higher costs at follow-up among those with problems on the Special Problems Rating Subscale that included this behaviour.

Clincal implications

Our results are important in helping to inform the practice and interventions that are most likely to help people with complex mental health needs progress in their rehabilitation. The finding that patients’ activity at recruitment into the study was associated with successful discharge/readiness for discharge at 12 months supports our decision to develop and test a staff training intervention aimed at promoting patient activity during other phases of the REAL study [26]. The association with social skills and successful discharge is also of interest. The evidence for the effectiveness of social skills training for people with schizophrenia has not been considered adequate for NICE [27] to recommend routinely offering it to people with schizophrenia. However, a meta-analysis of 22 trials of social skills training [28] found it to be associated with improvements in psychosocial functioning and negative symptoms, though problems with heterogeneity of methods and reporting of results limited the robustness of the findings [29]. Nevertheless, the Scottish Intercollegiate Guidelines Network guidance on the management of schizophrenia [30] states “social skills training may be considered for individuals diagnosed with schizophrenia who have persisting problems related to social skills.” Further studies are required to investigate the potential benefit of specific social skills training for people with complex mental health problems who are referred to rehabilitation services.

Our finding of a positive association between successful discharge and recovery orientated practice is of particular interest. Recovery orientated practice in mental health services is strongly encouraged by policy makers [31]. It incorporates a focus on therapeutic optimism and collaborative working with patients to agree together the goals of treatment and support, rather than the more traditional approach of a professional led treatment plan with the patient as passive recipient [32, 33]. Mental health rehabilitation services were early adopters of the recovery approach [34] and current commissioning guidance describes them as operating with this style and values [24]. One specific aspect of recovery orientated practice, namely the employment of ex-patients as members of the inpatient staff team occurred in 38 % of units (and one third of units across England in our national survey). The Recovery Based Practice domain of the QuIRC also includes many other aspects of care, including assessment of the degree to which collaborative care planning practices are employed and the therapeutic optimisim of the staff. We believe that our results may provide the first empirical evidence of the possible benefits of recovery orientated practice for people with complex psychosis.

We also identified factors associated with less chance of successful discharge/readiness for discharge. The greater the percentage of patients in the unit who had received CBT in the year before recruitment into the study, the less likely successful discharge was. Whilst this could be interpreted as suggesting a negative effect of CBT, there is strong evidence of its effectiveness in people with psychosis and it is recommended for treatment of this group [27]. A more likely explanation is that patients with the most complex needs, who are hardest to treat, are more likely to receive CBT as part of the range of interventions aimed at improving symptoms and functioning. This explanation concurs with the finding that patients who had been in hospital longer were less likely to achieve successful discharge/readiness for discharge. In other words, those with the most complex and treatment resistant symptoms tend to remain in hospital longer and are, perhaps, more likely to be offered more interventions over time. A possible alternative explanation is that patients who engaged with CBT developed greater insight into their mental health problems but required longer inpatient treatment as a consequence. However, we did not assess insight in this study and must stress that this possible association is purely hypothetical.

A history of fire setting was also associated with less chance of successful discharge/readiness for discharge and greater costs of care, although only 7 % of the cohort had such a history. Challenging and dangerous behaviours have previously been noted to make individuals difficult to discharge from hospital [5, 8]. Arson is an especially challenging behaviour and many supported accommodation providers are, understandably, reluctant to offer placements to people with this kind of serious risk history. Conversely, we found that a history of self-harm (which had occurred for 41 % of the cohort) was associated with a greater chance of successful discharge/readiness for discharge. This seems a rather paradoxical finding. Perhaps those who self-harm have less severe negative symptoms and are more motivated to act (albeit in a detrimental manner) than those with more severe negative symptoms whose level of function is so poor that it impedes community discharge. Self-harm may also indicate the presence of mood symptoms [35] which are generally associated with a better prognosis than negative symptoms alone. It should also be borne in mind that this factor included self-harm at any point in the person’s history, and these acts may have been many years earlier.

The fact that 14 % (1 in 7) of patients whom staff considered ready for discharge could not leave the unit because no suitable community accommodation was available is concerning. This represents an inefficient use of resources and needs to be addressed urgently to ensure that patients are supported in the least restrictive environment appropriate to their needs. More investment in community based supported accommodation is therefore required. This should include specialist accommodation for the small percentage of service users whose challenging behaviours such as a history of fire setting, impede move-on.

Strengths and limitations

Our study was only able to report associations between service and service user characteristics and outcomes. Since mental health rehabiliation services have been in place across England for many years, randomisation was not possible and there are no suitable comparison services that could be used for a case control design. The only feasible approach to evaluation therefore, was an observational study. Whilst our analyses were exploratory, we can have some confidence in our findings. We recruited a large sample from across most of the better performing inpatient mental health rehabilitation units in England. Our follow-up rate was excellent, with primary outcome data on successful discharge collected on over 90 % of our cohort. This is also a “hard” dichotomous outcome which does not rely on subjective opinion. Our decision to use staff rated outcomes also minimised the amount of missing data on patients’ social functioning at 12 month follow-up. However, inclusion of patient rated outcomes, such as quality of life and satisfaction with treatment and support, would have allowed us to report on a more comprehensive range of perspectives on the concept of “meaningful” clinical outcome.

Nevertheless, our results appear to support the therapeutic optimism that is enouraged in mental health rehabilitation services. The majority of patients in our study were successfully discharged to the community within our 12 month follow-up period (without readmission or community placement breakdown), despite the severity and complexity of their mental health problems that had led to their referral to these specialist services. Mental health rehabilitation services are therefore succeeding with an especially complex group and reducing the costs of care through their input. We found that successful discharge was associated with units operating an approach that incorporated a recovery orientation. Higher levels of patient activity and social skills were also associated with a greater chance of successful discharge. Patients with more complex needs and challenging behaviours (specifically, fire setting) were less likely to achieve successful discharge. Our findings suggest that further research is needed to identify effective interventions that enhance recovery orientated practice, patient activities and social skills in these settings.

Whilst our study was carried out in better performing services in England, the results have obvious relevance for lower quality units, not just in the UK, but in other countries where the quality of care may differ.

Ethics approval and consent to participate

The REAL study was approved by the South East Essex Research Ethics Committee (Ref. 09/H1102/45). All participants provided written informed consent to take part including their willingness for anonymised findings to be used in publications and reports.

Consent for publication

All authors approved the final version of the manuscript and agreed their accountability in ensuring that any questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Availability of data and materials

All data supporting our findings will be shared on request.