The use of routine outcome measures in two child and adolescent mental health services: a completed audit cycle

Outcome measures offer a window on the clinical effectiveness of interventions, providing important information to clinicians, managers, commissioners and service users. The importance of measuring outcomes within Child and Adolescent Mental Health Services (CAMHS) was recognised in England through the government’s NHS Outcomes Framework policy [1] and the National Service Framework (NSF) [2] for children and young people. The NSF suggests that the effectiveness of CAMHS interventions should be evaluated from multiple perspectives, including the clinician, parent/carer and where possible, the young person. The NSF states that this information is vital to improve future clinical work and additional administrative, Information Technology (IT) and clinical support should be available to enable CAMHS to use outcome measures.

To help support the implementation of routine outcome measurement (ROM) within CAMHS, the CAMHS Outcome Research Consortium (CORC) was created. The Consortium originally recommended the use of four outcome measures: the Strengths and Difficulties Questionnaire (SDQ) [3], the Children’s Global Assessment Scale (C-GAS) [4], the Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA) [5], and the Commission for Health Improvement-Experience of Service Questionnaire (CHI-ESQ) [6]. More recently, CORC also developed a pilot measure, the Goals Based Outcome (GBO) [7] which enables the client to monitor goal achievement following an intervention.

The outcome measures advocated by CORC hold specific advantages. As 'generic’ outcome measures, they are applicable across a wide range of clinical conditions seen in child and adolescent psychiatry and can be used routinely regardless of the client’s condition. Data can be collated at a service level to provide important information to commissioners on the effectiveness of services, and can be used clinically to inform both the clinician and the service user of any changes that may result from the intervention. HoNOSCA and C-GAS can quantitatively provide the clinicians opinion of patient functioning, whereas the SDQ and GBO offer the service user voice on their functioning. The CHI-ESQ also provides the service user with an opportunity to comment on their experience of the services. According to the Consortium guidelines, the SDQ, HoNOSCA and C-GAS should be completed (as a minimum) at baseline (time 1) and 6-month follow-up (time 2). The CHI-ESQ should be completed upon discharge from the service. Measures that are completed at baseline can be important for informing the assessment process. However, measures that are only completed at one time point cannot measure within-individual change which is an essential feature of outcome measurement, as such the completion of the same measure at follow-up is crucial to meet the criteria of outcome measurement.

Since 2011, the Consortium has been commissioned by the Department of Health to support the analysis of outcome measurements collated through the Children and Young People’s Improving Access to Psychological Therapies (CYP-IAPT; http://www.IAPT.nhs.uk), an initiative created to improve services for families and young people attending CAMHS by routinely assessing their opinion on the quality and experience of services. The initiative also recommends the use of the SDQ, GBO and CHI-ESQ as outcome measures (http://www.iapt.nhs.uk/silo/files/cyp-iapt-outcomes-summary.pdf).

The CYP-IAPT recommends the completion of the SDQ and also an anxiety and depression scale (RCADS: Revised Child Anxiety and Depression Scale) [8] at initial assessment. At each subsequent session they suggest the use of brief scales such as the GBO and ORS (Outcome Rating Scale) [9] to measure functioning and the SRS (Session Rating Scale) [10] to assess client satisfaction. At the review stage the initiative advocates the use of the same measures that were completed at assessment (SDQ, RCADS) alongside a CHI-ESQ.

The research on implementing outcome measures in UK child and adolescent psychiatry indicates a relatively low uptake of these measures [11]. Johnston and Gowers’ [11] survey revealed that less than 30% of the 186 CAMHS teams that responded had routinely used ROM, with the SDQ and HoNOSCA being the most commonly used measures. Baruch and Vrouva [12] found a higher uptake of outcome measures for young people attending a psychotherapy unit where completion of self-reported measures was 94% at baseline, dropping to 35% at 3-months and even less at 6 and 12-month follow-up. In a recent multi-method study, Batty et al. [13] conducted a survey and workshop to assess stakeholders’ opinions of routine outcome measurement in CAMHS, and also carried out a case-note audit documenting the current use of outcome measures in three community CAMHS teams. In support of Johnston and Gowers [11] they also found a poor uptake of outcome measures.

Common reasons for the lack of completion of outcome measures in clinical practice reflect constraints on both time and resources [11, 13–15]. The lack of timely feedback from completed outcome measures has also been shown to decrease clinicians’ motivation to use them, rendering them a 'tick box’ exercise used to meet targets but with little clinical utility [13]. Other noted barriers to the uptake of ROM include a lack of training [16, 17], some clinicians unwillingness to use a systematic approach to data collection [18] and concerns about how the data will be used by managers and commissioners [19, 20].

This study reports on the uptake of ROM across two CAMH services in the East Midlands region of England (Derbyshire and Nottinghamshire) and presents the findings of a re-audit and the baseline audit reported in Batty et al. [13]. Our aim is to provide an update of the current use of outcome measures and assess whether recent government strategies, (such as the increase in the provision of administration support to collect these measures) and local initiatives (such as the Commissioning for Quality and Innovation (CQUIN) target), alongside early CYP-IAPT trials and previous research has improved the uptake of outcome measures. This research was conducted as part of the National Institute for Health Research (NIHR) Collaborations for Leadership in Applied Health Research and Care- Nottinghamshire, Derbyshire and Lincolnshire (CLAHRC-NDL).

Patient records were examined in three tier 3 (specialist or highly specialist) CAMHS clinics across two Trusts: Nottinghamshire Healthcare NHS Trust (NHCT) and Derbyshire Healthcare NHS Foundation Trust (DHCFT). Two clinics participated in Nottinghamshire, and one in southern Derbyshire.

Case-notes were accessed by a member of the Trust’s clinical team. NHCT originally signed up to the CORC scheme but unsubscribed in 2012. The Trust introduced a Commissioning for Quality and Innovation (CQUIN) target in April 2011 to complete any one of the following four measures on the electronic record system: SDQ, HoNOSCA, C-GAS and GBO at time 1 (initial assessment) and time 2 (6 month follow-up, or discharge if sooner). CQUIN targets allow commissioners to reward excellent performance by linking the healthcare providers’ income to the achievement of local improvement goals. DHCFT also signed up to the Consortium scheme several years ago and were in the first wave of the CYP- IAPT provider organisations in 2011/12, but have no specific commissioning targets relating to the completion of outcome measures.

The original audit was conducted between December 2010 and January 2011 with the re-audit taking place between December 2012 and February 2013. The same method and protocol were used for both audits. For the original audit, 20 case notes were examined in Derbyshire and North Nottinghamshire, and 21 in South Nottinghamshire. For the re-audit, 20 case notes were included from each clinic that participated in the original audit. In Nottinghamshire (north and south) there were 1250 open referrals, in southern Derbyshire there were 191.

Ethical approval for the study was granted by the local Research Ethics Committee and Research and Development Departments of NHCT and DHCFT.

We compared the use of ROM in CAMHS from an audit conducted in 2011 [13] with a re-audit conducted in 2012/2013 to assess for any changes in the use of outcome measures as a result of recent research, government and commissioning strategies. Our findings reveal a significantly greater uptake of HoNOSCA, SDQ and C-GAS, which are measures advocated by a national Child and adolescent mental health services Outcome Research Consortium (CORC), and crucial to fulfil targets outlined in local commissioning policy (CQUIN). We also noted an increase in the repeated use of outcome measures.

In line with the findings from the original audit [13], HoNOSCA was the most frequently used measure in the re-audit, followed by C-GAS. Additionally, both the single and repeated use of these measures had significantly increased since the original audit. As reported by Batty et al. [13], HoNOSCA has a longstanding history and expectation of use within these Trusts which is likely to explain its high completion rate. As both HoNOSCA and C-GAS are clinician-completed measures it is easier for clinicians to ensure that these measures are completed, in comparison to measures completed by the service user or caregiver (e.g. SDQ, GBO, CHI-ESQ) which often involve more administrative support and the co-operation of the service user. The improved completion rates for clinician rated outcome measures offer the service user, clinician and managers with quantifiable evidence of any change that may resulted from the intervention. This allows all stakeholders to assess the effectiveness of the service and the individual may assess the benefit of the treatment received. However, the comparatively lower completion of service user measures demonstrates that the perspective of change as recorded by the service user is under-reported. It is important that change from both the perspective of the clinician and service user is recorded to fully understand the effectiveness of any intervention. This is particularly the case as clinician completed measures can be susceptible to reporting bias, such as over reporting the extent of improvement [22].

The re-audit revealed a greater use of combined measures in comparison to that found in the original audit. As shown in Figure 1, in the original audit HoNOSCA and C-GAS were the most common combination of measures, whereas in the re-audit, HoNOSCA, SDQ, and C-GAS were the most common grouping of measures, with this combination appearing in over half of all case notes. This suggests that clinicians are in greater agreement regarding which combination of outcome measures to use. The increased uptake of the SDQ may also reflect clinicians’ positive attitudes towards this measure, with previous research showing that they value this as a measure of service users’ opinion [13]. Additionally, the increase in provision of administration time for outcome measures may have been influential for the completion of these service-user completed measures which involve extra burden in terms of posting-out and collecting questionnaires. In contrast, other CORC advocated measures such as the GBO and CHI-ESQ were rarely found. Again, it may be that this represents difficulties in getting measures completed by the service user or, given that these measures are relatively new with less established psychometric properties and history of use within CAMHS, clinicians may be unwilling to engage with these measures. Previous research has noted concerns about the scientific quality of measures and lack of knowledge in how to use and interpret a measure impedes clinicians’ likelihood to use a measure [11, 14, 23]. It is also possible that clinicians feel that a combination of the clinician completed measures alongside one service user measure, such as the SDQ which has established validity and reliability [24–26], is satisfactory in gaining an impression of current functioning.

The significant increase in both the single and repeated use of different outcome measures since the original audit may have resulted from factors other than increased administration time. Additionally, initiatives such as CORC have promoted greater awareness or the use and type of outcome measures, specifically, in the use of generic rather than condition-specific measures that allow comparisons to benchmarks which should lead to improvements in practice [27]. The influence of the recent IAPT initiative is also already seen in DHCFT, whereby the uptake of the IAPT advocated RCADS was found in some case notes. Although routine use of all CYP-IAPT scales was only introduced after the re-audit, the initial year of the scheme involved training clinicians on the clinical usefulness and importance of completing ROMs, thus establishing a managerial expectation that they would be completed. However, it is interesting to note that NHCT withdrew their CORC membership after the original audit but increased their use of outcome measures, suggesting that any practical support provided by CORC was less influential.

We have also speculated that the collection of outcome measures may be driven by CQUIN targets, but DHCFT did not have any commissioning strategies associated with the completion of outcome measures (i.e. CAMHS outcomes CQUIN); yet comparable results were found across the sites with regard to CORC measures. Thus, it may be that clinicians’ willingness to use a measure may be driven as much by knowledge and awareness of measures as it is by commissioning policies. Since the original audit, the research organisation, Collaborations for Leadership in Applied Health Research and Care – Nottinghamshire, Derbyshire, Lincolnshire (CLAHRC-NDL) has conducted significant work to promote the use of outcome measures across the East Midlands. Such work includes seconding 'Diffusion Fellows’ and other local champions from NHS partners to translate and disseminate knowledge from research studies into practice, holding conferences and seminars for local clinicians and publishing findings of the original audit in simple summary 'bites’ for clinicians and managers. Given that previous research has highlighted local champions as providing a key role in promoting outcome measures [28], it is likely that CLAHRC activities have also partially driven this change across the two NHS Trusts.

Whereas the original audit [13] and previous research [11] noted very little repeated use of the same measure, we found that approximately 60% of case-notes contained repeated use of the same measure, compared to 30% in the original audit. Although this shows an improvement in ROM, given that almost half the case-notes still contained only a single use of a given measure, there is further work needed to improve the rate of outcome measurement in CAMHS.

As the option of entering CORC measures to the electronic records system had not been developed at the time of the first audit, we cannot make comparisons regarding the number of measures that are being recorded electronically. However, findings from the re-audit demonstrate that the majority of measures are being recorded electronically. This is important for a care system where multiple professionals with specialised knowledge may be involved in care delivery in different geographical areas [29]. It is possible that the option of electronic inputting of data may have contributed to the increased use of outcome measures, allowing clinicians to quickly input clinician-completed outcome measures (HoNOSCA and C-GAS) without having to find paper-versions. Electronic records offer the opportunity of better access to patient information, with the premise that the greater the availability of high quality information the more able the clinician is to care for the patient [30]. Furthermore, the use of an electronic system allows opportunity for the development of a report system that graphically represents changes in outcome scores over time. This system would provide clinicians with real-time feedback on their client’s progress, thus increasing the clinical utility of outcome measures [13]. Additionally, this type of report could allow data to be aggregated at team or service level to inform managers and commissioners to enable benchmarking to comparable services [31].

Although the re-audit has shown there is a significant increase in the uptake of ROM in child and adolescent psychiatry, it has also highlighted the need for further improvement, particularly with regard to repeated use of the same measure and measures completed by the service user (e.g. SDQ, GBO, CHI-ESQ). In order for an outcome measure to assess any changes that may have resulted through an intervention it is imperative that the same measure is used at baseline and at least once thereafter. To reduce burden on clinicians future outcome measure initiatives may wish to consider reducing the number of clinician completed measures required. Given that HoNOSCA has been shown to have better reliability and be more informative than C-GAS [32] it may be prudent to only complete HoNOSCA. However, C-GAS provides information about the level of functioning of the service user in the previous month [4] across all conditions, incorporating elements of a multi-axial assessment [33]; therefore it is considered a valuable complement to HoNOSCA in research [34] and clinical practice. To improve the completion of service user completed outcome measures such as the SDQ, new technologies could be implemented to facilitate their use. For example, the measures could be completed on a tablet PC in the waiting room prior to each clinic session, (a system currently being rolled out by the CYP-IAPT initiative and separately a trial of electronically completed measures which is being evaluated by the CLAHRC-NDL). The reports from these measures could be fed straight back to the clinician online, producing real-time feedback that does not rely on service users having to remember to complete and post questionnaires prior to their clinic appointment. Truman et al. [35] report on a computer-based SDQ and found significantly more user satisfaction with a computer version in comparison to the paper-based version. This kind of measure requires significant investment in technical adaptation to ensure integration with electronic patient records and would require managerial understanding and commitment to proceed. This 'session-by-session’ monitoring would overcome difficulties in getting follow-up measures due to treatment drop-out, or clinic appointments that are not scheduled around the 6-month follow-up. Furthermore, this regular monitoring may be more sensitive to change and may also allow clinicians to modify their intervention strategy earlier on if they felt sufficient progress was not being made [36].

The comparison of two audits has offered a valuable insight into the improvements of ROM within child and adolescent psychiatry which may have resulted from greater Trust support and initiatives such as CLAHRC-NDL research, CORC and CYP-IAPT that actively promote the use of outcome measures. However, our findings are limited to two NHS Trusts; as such caution should be taken when generalising the findings to other Trusts located in different geographical regions. Nethertheless, the comparison of two different Trusts allowed for the assessment of local service drivers and priorities and their impact on outcome measure completion. Given that the aim of this research was to document the evidence-base for ROM in CAMHS we did not assess clinicians’ opinions as to which factors were influencing their use of specific outcome measures in the re-audit. However, we have inferred possible barriers based on well documented findings from previous research [13, 14, 16, 17, 20].

A comparison of two audits has revealed an increase in the use of outcome measures within CAMHS, particularly for clinician-completed measures. The possible increase in clinician awareness and training in outcome measures, alongside dedicated administration support may have facilitated the process of ROM. It is important that initiatives continue to increase clinicians’ awareness of the importance of measuring outcome within CAMHS, with education specifically focussing on the need to assess whether patients have benefited from the result of the intervention, in line with more physically-based health services. Further initiatives are also needed to improve the repeated use of measures and service user completed measures. As described, innovative models of session-by-session monitoring on tablet PCs could further increase clinician and service-user engagement with outcome measures by providing instant, clinically-useful feedback.