Skip to main content
Download PDF
  • Research article
  • Open access
  • Published:

Prioritizing suicide prevention guideline recommendations in specialist mental healthcare: a Delphi study

BMC Psychiatry volume 20, Article number: 55 (2020) Cite this article

Abstract

Background

The Delphi technique is a proven and reliable method to create common definitions and to achieve convergence of opinion. This study aimed to prioritize suicide prevention guideline recommendations and to develop a set of quality indicators (QIs) for suicide prevention in specialist mental healthcare.

Methods

This study selected 12 key recommendations from the guideline to modify them into QIs. After feedback from two face-to-face workgroup sessions, 11 recommendations were rephrased and selected to serve as QIs. Next, a Delphi study with the 11 QIs was performed to achieve convergence of opinion among a panel of 90 participants (23 suicide experts, 23 members of patients’ advisory boards or experts with experiences in suicidal behavior and 44 mental healthcare professionals). The participants scored the 11 QIs on two selection criteria: relevance (it affects the number of suicides in the institution) and action orientation (institutions or professionals themselves can influence it) using a 5-point Likert scale. Also, data analysts working in mental healthcare institutions (MHIs) rated each QI on feasibility (is it feasible to monitor and extract from existing systems). Consensus was defined as 70% agreement with priority scores of four or five.

Results

Out of the 11 recommendations, participants prioritized five recommendations as relevant and action-oriented in optimizing the quality of care for suicide prevention: 1) screening for suicidal thoughts and behavior, 2) safety plan, 3) early follow-up on discharge, 4) continuity of care and 5) involving family or significant others. Only one of the 11 recommendations early follow-up on discharge reached consensus on all three selection criteria (relevance, action orientation, and feasibility).

Conclusions

The prioritization of relevant and action-oriented suicide prevention guideline recommendations is an important step towards the improvement of quality of care in specialist mental healthcare.

Peer Review reports

Background

In the Netherlands, about 40% of all people who die by suicide are in specialist mental health care [1]. The implementation of suicide prevention guideline recommendations into routine care appears to be an effective strategy to prevent suicides in MHIs. A large-scale study in the United Kingdom showed a significant reduction in suicide rates after implementation of 16 service improvements (including training of clinical staff, policy on the follow-up of discharged patients and ward safety) in MHIs [2]. In the Netherlands, the multidisciplinary guideline for the diagnosis and treatment of suicidal behavior was published in 2012 [3]. This guideline was developed to optimize the care for patients with suicidality in mental healthcare. However, its uptake by the field is varying, resulting in different suicide prevention policies and practices within and between MHIs in the Netherlands [4, 5].

To optimize guideline implementation and reduce the suicide rate in Dutch MHIs, 113 Suicide Prevention (the Dutch expertise center on suicide prevention and lifeline) formed a Suicide Prevention Action Network (SUPRANET) in mental healthcare (www.supranetggz.nl). SUPRANET is a confidential learning network of at present 16 specialist MHIs in the Netherlands. SUPRANET aims at optimizing the quality and safety of care to enhance suicide prevention. This network collects data on suicide and suicide attempts, provides biannual benchmark feedback reports to participant organizations and organizes meetings for the exchange of best-practices [6].

To improve the quality of care and prevent suicides as well as eliminate discrepancies in specialist mental healthcare, it is of great importance to prioritize the guideline recommendations and to define measurable elements designed to evaluate aspects of the quality of care [7,8,9,10]. These QIs should be relevant, actionable, reliable, show room for improvement and data collection of the indicator should be feasible [11]. Using QIs to monitor changes in adherence to the suicide prevention guideline in specialist mental healthcare is an essential step towards the delivery of evidence-based care [12].

Because all patients with an increased risk for suicidality should receive evidence-based care within MHIs, the literature should try to underpin the effectiveness of every QI and its relation to reduced suicide (attempt) rates. Up until now, the literature examined the effectiveness of several guideline recommendations [3]. A variety of pharmacological and psychotherapeutic interventions have repeatedly been found effective in the treatment of suicidality [13,14,15,16,17]. For example, a study of Linehan et al. [18] found evidence for the effectiveness of dialectical behavior therapy (DBT) in decreasing the number of suicide attempts as well as reducing the number of hospitalization visits for suicide ideation in women with a borderline personality disorder [19]. To our knowledge, however, not all recommendations from the suicide prevention guideline have been extensively researched in the literature [3]. For example, safety planning is often informed by research and clinical practice as an important guideline recommendation but does not yet have a body of research to support it. Fortunately, the evidence is building. A randomized controlled trial (RCT) by Bryan et al. [20] found a positive association between crisis response planning (an intervention related to safety planning) and reduced suicide attempt rates compared to the use of contracts for safety (e.g., which states that the patient with suicidality should not engage in suicidal behavior during a crisis) in US soldiers. Although several studies have been published on the effectiveness and use of safety planning [21], more research is still needed. A recent publication by Brent, Oquendo, and Reynolds [22] emphasized that formulating a safety plan is one of the seven key elements to treat patients with suicidality effectively.

As suggested by the review of Zalsman et al. [13], no single suicide prevention strategy is way more effective compared to the others. Multiple relevant and action-oriented key recommendations should therefore be implemented at the same time. Also, While et al. [23] showed that mental healthcare services who implement multiple strategies (seven to nine recommendations) at the same time show the most significant reduction in the number of suicides compared to those implementing fewer recommendations.

This study aimed to prioritize the suicide prevention guideline recommendations and to develop a set of relevant and actionable QIs for suicide prevention in specialist mental healthcare. The Delphi technique was used to create common definitions and terminology, and to achieve convergence of opinion among the participants. Participants of this study were suicide experts, health care professionals, and experts with experiences in suicidal behavior and members of patients’ advisory boards. Criteria for selection were 1) relevance (it affects the number of suicides in the institution) and 2) action orientation (the institutions or professionals themselves can influence it). In the last step, data analysts scored the QIs on 3) feasibility to monitor and extract them from existing systems. The Delphi technique is a reliable method in selecting QIs and uses a structured, iterative process of collecting knowledge from a group of experts with the primary goal of reaching consensus [24,25,26].

Methods

As part of the SUPRANET study [6], a Delphi study was done to prioritize the guideline recommendations and to select and standardize the terminology of QIs. This study has been approved by the Central Committee on Research Involving Human Subjects in the Netherlands (CCMO) and does not fall under the scope of the Medical Research Involving Human Subjects Act (WMO). The CCMO states that: “In general, research with human subjects only falls under the Medical Research (Human Subjects) Act (WMO) if there is an infringement of the physical and/or psychological integrity of the subject” (https://english.ccmo.nl/investigators/legal-framework-for-medical-scientific-research/your-research-is-it-subject-to-the-wmo-or-not).

For this study, the SQUIRE 2.0 checklist (Standards for Quality Improvement Reporting Excellence) was used as a reporting guideline [27]. The procedure for the development of the QIs is shown in Fig. 1 and described in the procedure section below.

Fig. 1
figure 1

Flowchart of Delphi study with procedure for QI development

Full size image

Procedure

The procedure was conducted in two phases:

  • Phase one: Development and selection of guideline recommendations and QIs

From the 49 key recommendations described in the suicide prevention guideline, an inventory of potential recommendations was selected (Fig. 1). These key recommendations were selected based on the possibility to operationalize them on a patient level. Furthermore, it was preferred that there was an evidence-based association between the key recommendations and reduced suicide (attempt) rates. This procedure resulted in a selection of 12 key recommendations from the guideline.

The selected key recommendations were modified into 12 process QIs (on a patient level) by describing them into detail, including definitions, numerators, and denominators and target values (see Additional files). Next, two small multidisciplinary workgroups of healthcare professionals (workgroup one; n = 16; workgroup two; n = 15) discussed the list of 12 QIs. During three sessions of one hour each, participants in both workgroups were instructed to discuss the 12 proposed QIs, based on their knowledge and expertise in the field. Furthermore, the participants appraised each QI individually with an overall score, reflecting whether the study should reject the QIs or rephrase them.

As illustrated in Fig. 1, participants rejected two QIs the Collaborative Assessment & Management of Suicidality (CAMS) and the Chronological Assessment of Suicide Events (CASE approach) due to low feasibility. Also, the participants in the workgroups argued that both indicators are not yet sufficiently implemented into routine care. The QI social connectedness was rejected by the participants due to a lack of scientific evidence. Also, the participants stated that social connectedness showed much overlap with other QIs and categorized this as a structural QI instead of an indicator that should be measured on a patient level. The numerator of the QI involvement of family or significant others was split into two components: A) there is contact with family or significant others and B) contact person is registered in patients’ electronic medical record (Additional files). Next, one QI was split into two QIs because the participants stated that the QI EHealth actually contains two important measurable components: 1) active usage and 2) availability of EHealth (Additional files). Finally, the participants also split the QI continuity of care: members of both workgroups agreed to divide this indicator into two QIs continuity of care and follow-up after discharge. Between each workgroup session, QIs were further clarified by the first author (KS) in terms of their clarity and definition. This resulted in a final set of 11 defined and operationalized QIs (Additional files). After final approval from the participants in both workgroups, the Delphi study was performed (phase two).

  • Phase two: Rating of guideline recommendations and QIs (Delphi study)

The Delphi study consisted of two rounds:

Round one: Rating QIs on relevance and action orientation

To achieve convergence of opinion, we developed an anonymous electronic survey with the 11 selected key recommendations and their QIs (Additional file 1). All MHIs participating in the SUPRANET network were involved in this study. Participants working in the field of suicide prevention were recruited from each MHI. Participants were approached in the first round via e-mail to complete the survey (Additional file 1) by following a web link. Written consent of participants was given by filling in the survey.

A total of 90 participants recruited from each MHI (23 suicide experts, 23 members of patients’ advisory boards or experts with experiences in suicidal behavior and 44 health care professionals), filled in the survey. They rated the QIs independently on two aspects: relevance (it affects the number of suicides in the institution) and action orientation (institutions or professionals themselves can influence it). All participants rated the QIs on both aspects using a 5-point Likert-scale (1 = strongly disagree, 2 = disagree, 3 = neither disagree or agree, 4 = agree and 5 = strongly agree).

Round two: Rating QIs on feasibility

In the second round, members with specific expertise on data such as data analysts working in MHIs (n = 6), independently rated the QIs on feasibility (is it feasible to monitor and extract from existing systems) using an online survey (Additional file 2). The QIs were scored on a 5-point Likert scale (1 = not feasible at all, 2 = not feasible, 3 = neither not feasible or feasible, 4 = feasible and 5 = completely feasible). After both Delphi rounds, outcomes were analyzed, resulting in a specific set of relevant, action-oriented and feasible QIs.

Data analyses

Analysis of both rounds of the Delphi study yielded consensus results by computing medians, and percentage of consensus for each item to determine which QIs achieved positive consensus. Data of respondents were anonymized prior to analysis. Agreement percentages were calculated for each item by assessing if at least 70% of the participant response rates were within the range of the median scores of four and five. The level of consensus achieved concerning the 11 QIs included in the first round was assessed using the following cut-off score [24, 28]:

  1. 1.

    Consensus = median score of four or higher on both selection criteria (relevance and action orientation) with > 70% consensus.

  2. 2.

    No consensus = median score of four or higher with < 70% consensus for one of the selection criteria (relevance or action orientation).

  3. 3.

    Not suitable = median score of three or lower for one of the selection criteria (relevance or action orientation).

Median scores and percentage of consensus was analyzed between the three groups of participants: 1) suicide experts (n = 23), 2) members of patients’ advisory boards and experts with experiences in suicidal behavior (n = 44) and 3) health care professionals (n = 23). Analyses were performed with SPSS version 24.0.

Results

Participants

The Delphi survey was sent to 154 experts by e-mail, and a total of 90 experts filled in the online questionnaire (response percentage of 58%). Of the 90 participants, 23 people (25.6%) were suicide experts, 23 (25.6%) were members of patients’ advisory boards or experts with experiences in suicidal behavior, and 44 (48.9%) were healthcare professionals. Almost all participants were from the Netherlands (n = 88), two participants lived in Belgium. From the participants that filled in the Delphi survey (N = 90), the mean age of the participants was 48.6 (SD = 11.39) years, of which 58.9% (n = 53) was female. The mean years of experience of healthcare professionals in Dutch mental healthcare was 18.10 years (SD = 9.73).

Round one: Rating  QIs on relevance and action orientation

In the first round of the Delphi study, participants reached a consensus regarding the relevance for eight of the 11 QIs, and five of the 11 QIs were rated as action-oriented (Table 1). The participants scored the following five QIs that belonged to the guideline as both relevant and action-oriented: 1) screening suicidal thoughts and behavior, 2) safety plan, 3) early follow-up on discharge, 4) continuity of care, and 5) involving family or significant others. As Table 1 shows, between-group differences for relevance and action orientation were found for several prioritized QIs (screening, continuity of care, early follow-up on discharge, and involving family or significant others). Participants reached consensus on relevance, and action orientation for QIs highlighted in bold.

Table 1 Percentage of consensus for both relevance and action orientation (round one)
Full size table

No consensus was reached on relevance and action orientation for three QIs: 1) availability of EHealth, 2) active usage of EHealth, and 3) evidence-based medication. Further, 1) structural diagnosis, 2) waiting list duration, and 3) evidence-based psychotherapy achieved consensus for relevance but were not rated as action-oriented by the participants.

Round two: Rating QIs on feasibility

During the second round of the Delphi study, data analysts (n = 6) rated the QIs on feasibility (Table 2). The percentage of consensus and median scores for the rated QIs on feasibility are shown in Table 2. When looking at the consensus cut-off score, participants rated only one of the 11 QIs as feasible. Early follow-up after discharge was prioritized as the only relevant, action-oriented, and feasible QI. For the QI waiting list duration, participants reached borderland consensus. As for the other nine QIs, no consensus on feasibility was reached. Both rounds of the Delphi study resulted in a final list of relevant, actionable and feasible QIs derived from the guideline. These results are presented in Table 3.

Table 2 Percentage of consensus and median scores for feasibility (round two). Percentage in bold achieved consensus
Full size table
Table 3 Final list of relevant, actionable and feasible quality indicators derived from the guideline
Full size table

Discussion

This study aimed to generate a set of relevant, action-oriented, and feasible QIs derived from the guideline by using a Delphi method. Only one QI was rated by six Dutch data analysts as feasible to monitor and register in MHIs in the Netherlands and reached consensus on all three criteria (relevancy, action orientation, feasibility). This QI was early follow-up on discharge. According to the literature, the risk of suicide is three times higher in the first week after discharge from a psychiatric facility [29] and remains significantly higher during the next few months [30, 31] or even years [32]. While the study of Zalsman et al. [13] reported that there is insufficient evidence for the efficacy of supportive contacts after discharge from the emergency department (ED), a large UK-study of Kapur et al. [2] found an association between rapid follow-up contact after inpatient discharge and lower suicide rates in mental healthcare services. Also, Luxton et al. [33] argued that brief supportive contacts by phone, text, postcards, or letters could be effective in reducing suicide (attempts) during high-risk periods (i.e., hospitalization or ED visits). Even though more high-quality RCTs are still needed to examine its impact further, numerous studies [14, 22, 34, 35], quality standards and suicide prevention guidelines [3, 36] repeatedly recommend improving early follow-up contact after discharge. However, whether a QI is rated as feasible will differ between institutions and countries.

Following the total group of experts, consensus on relevance (it affects the number of suicides in the institution), and action orientation (the institutions or professionals themselves can influence it) was reached on the following four QIs:

  1. I.

    Screening of suicidal thoughts and behavior

Based on our results, the QI screening of suicidal thoughts and behavior was rated as a high priority. While some clinicians hold a belief that asking about suicide could trigger a patient into a suicidal act [37], a recent review found no iatrogenic effects [38]. An extensive systematic review of Zalsman et al. [13] reported inconclusive results on the costs and effects of routine screening and its association with reduced suicide (attempt) rates. Results from other studies also found that screening instruments are not very accurate in identifying the risk of suicide attempts or suicide death in individuals [39, 40]. A study by Coffey [41], however, showed that routine screening could be a useful and feasible method if screening results are regularly fed back to care teams and if it is included in the registration systems of MHIs. Although more empirical research on the effects of routine screening is needed, it is perceived by the literature as a very helpful strategy [22, 34, 35, 42]. To effectively treat individuals whose risk of suicide is high, it is very important to identify those individuals. Although evidence shows that predicting suicide with certainty is rather complex, it is not necessary to perfectly predict suicide in order to intervene effectively. For example, the National Action Alliance for Suicide Prevention [42] argues that by identifying individuals with an elevated risk, it allows mental healthcare professionals to detect those high-risk individuals who could then be treated with adequate and effective interventions.

  1. II.

    Safety plan

Safety plan was rated as an essential QI for mental healthcare. The Dutch multidisciplinary suicide prevention guideline and care quality standards recommend regular and collaborative safety planning for all patients at risk of suicide [3, 43]. Also, safety plans are embedded in cognitive behavioral therapy (CBT) for suicide prevention [44]. Literature shows that there is some empirical evidence on the effectiveness of safety plans in reducing suicide (attempt) rates. For example, an RCT study of Bryan et al. [20] examined the impact of crisis response safety planning on suicide (attempts) among US soldiers at high-risk of suicide. The authors found a significant reduction of suicide ideation, fewer hospitalization days, and a 76% reduction in suicide attempts. These results are in line with a cohort comparison study of Stanley et al. [21], where a significant association between a Safety Planning Intervention (SPI) and reduced suicidal behavior was found in veteran hospitals. It should be noted, however, that this study design is limited since both intervention and control emergency department sites were matched instead of being randomized. Furthermore, both studies were performed in a veteran population. The results from both studies on the impact of safety plans are promising, but more RCT studies on its impact, also in other study populations, are highly needed.

  1. III.

    Continuity of care

Consensus was achieved for continuity of care as a relevant andaction-oriented QI. In daily practice, it often happens that more than one clinician treats a patient at risk of suicide in a short period of time [3]. Transfer between clinicians, settings, or organizations could affect the continuity of care and is a well-known risk factor for patients at risk of suicide [45]. It is, therefore, important that clinicians collaborate and inform each other to ensure these patients receive the care they need. A few years ago, a report was published containing information about how continuity of care should be organized in Dutch mental healthcare [46]. This report aimed to stimulate and improve continuity of care by formulating recommendations (i.e., following suicide prevention training) to healthcare professionals (including general practitioners, medical specialists, emergency doctors). The literature highly recommends minimizing discontinuities in care for patients at risk of suicide during critical phases (i.e., transfers, post-discharge) [3, 34, 35]. However, more empirical research on its impact is needed. Fortunately, the scientific  evidence is building. For example, the study of Kapur et al. [2] found a significant association between the implementation of policies aiming at optimizing continuity of care for patients at risk of suicide in mental healthcare services and a reduction in the number of suicides . The importance of close communication between clinicians and across mental healthcare settings is also emphasized by other literature [22, 47]. Other studies should try to replicate the results of Kapur et al. [2] to further determine the impact of continuity of care on reduced suicide rates in mental healthcare.

  1. IV.

    Involving family or significant others

Although the impact of involving family, carers or friends on suicide rates is rather understudied in the literature, this strategy on itself is highly recommended by the Dutch multidisciplinary suicide prevention guideline and various quality standards [3, 48]. Relatives can play an essential role in the prevention of suicide, but only if they are capable of supporting the mental healthcare services in the early detection and management of family members at risk [49]. They can provide valuable information to clinicians that could be helpful for the patients’ treatment. The importance of actively involving and informing family members is also highlighted in other studies [50, 51]. However, it is important to take into account that patients should always be allowed to say whether they would like their family or friends to be involved in their treatment or not [48]. A clinician should bear in mind the patients’ expressed wishes and views in relation to sharing information with their family, carers, or friends. On the contrary, relatives of a patient should be supported and provided with information as well. Based on a report of the department of Health and Social Care in England, relatives and carers have repeatedly raised their concerns about mental healthcare services who seem reluctant to take information from families or give them information about a patients’ suicide risk (https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/271792/Consensus_statement_on_information_sharing.pdf). It is of great importance to address these concerns, given the fact that the collaboration between a therapist and patients’ relatives is one of the components which can lead to better quality and safety improvement in specialist mental healthcare [34].

Not prioritized guideline recommendations

Some guideline recommendations and their QIs were not endorsed as priorities, including evidence-based medication and psychotherapy for patients in specialist mental healthcare. This is surprising given the fact that both medication (lithium, clozapine, ketamine) and psychotherapy have shown promising results in reducing suicidality among patients in mental healthcare [13, 15,16,17].

It is possible that because medication and psychotherapy are already embedded in routine care, both recommendations were considered to be of less priority. Furthermore, participants in this study reported that medication and psychotherapy could not be influenced directly or changed by the MHIs or the healthcare professionals themselves, leading to a lower score on action orientation. Between-group differences were found for several guideline recommendations. For example, the expert group with members of patients’ advisory boards or experts with experiences in suicidal behavior rated screening as highly relevant and action-oriented, whereas the other two groups reached borderland consensus for this guideline recommendation. Also, the suicide expert group did not reach an agreement on action orientation for several recommendations and their QIs: 1) early follow-up on discharge, 2) continuity of care, and 3) involving family or significant others. This is in contrast with the outcomes of the other two participating expert groups, who did succeed in achieving consensus on action orientation for the same recommendations.

Guideline implementation is intended to result in a higher quality of mental healthcare, eventually leading to lower suicide rates. To achieve this goal, scientific evidence suggests that an optimal approach is required for implementing a combination of multiple relevant and action-oriented guideline recommendations at the same time. For example, a large UK-study examined the association between the implementation of key mental health service recommendations and suicide rates [23]. They found that services that had implemented seven to nine recommendations had a significantly lower suicide rate than those implementing fewer recommendations [23]. These findings suggest that the effort of implementing recommendations in mental healthcare services can affect suicide rates.

Strengths and limitations

A key strength of this study is that experts from MHIs were continuously involved in both the development and selection of guideline recommendations and their QIs. Another strength was the breadth of expertise of the participants who participated in the Delphi study. Also, this survey sample (N = 90) was notably larger compared to other Delphi studies in the field of mental health [28]. The size of a Delphi panel is generally under 50, although more have been employed [25]. As for both rounds, the survey response rate was good overall. Nevertheless, there are some limitations. First, the participants in this study mainly came from the Netherlands. The established set of QIs might not match MHIs in other countries, and this applies in particular to the feasibility of the QIs. Second, the survey did not offer an opportunity for the 90 experts to suggest QIs that were not derived from the suicide prevention guideline.

Conclusions

This study stands to contribute to the scientific literature on prioritizing relevant and action-oriented suicide prevention guideline recommendations to improve the quality of care in specialist mental healthcare. The results of this study are an important step towards the use of a new set of relevant and actionable QIs that are clearly defined, operationalized, and contain target values, making them appropriate for benchmarking and monitoring guideline implementation in specialist mental healthcare. If needed, other countries can tailor the selected QIs to one’s own (specialist) mental healthcare setting.

The prioritized QIs will be used to monitor the degree of guideline implementation within the specialist MHIs [6]. The five guideline recommendations selected as relevant and action-oriented might be of most importance for suicide prevention when implemented altogether. The SUPRANET study [6] will evaluate the implementation process of the prioritized recommendations in MHIs in the Netherlands. When proven to be effective, MHIs should include the prioritized recommendations into their policy for suicide prevention.

Availability of data and materials

The datasets used and/or analyzed during this study will be made available after approval from the corresponding author or the SUPRANET board on reasonable request.

Abbreviations

CAMS:

Collaborative Assessment & Management of Suicidality intervention

CASE:

Chronological Assessment of Suicide Events

MHI:

Mental healthcare institution

PHQ:

Patient Health Questionnaire-9

QI:

Quality indicator

SQUIRE:

Standards for Quality Improvement Reporting Excellence

SUPRANET:

Suicide PRevention Action NETwork

References

  1. Healthcare Inspectorate. Overzicht van het aantal suïcides en suïcidepogingen met ernstig letsel tot gevolg binnen de ggz-instellingen. 2019 Retrieved from https://www.igz.nl/onderwerpen/publieke-en-geestelijke-gezondheidszorg/geestelijke_gezondheidszorg/cijfers_over_suicides_ggz/. Accessed Oct 2019.

  2. Kapur N, Ibrahim S, While D, Baird A, Rodway C, Hunt IM, et al. Mental health service changes, organisational factors, and patient suicide in England in 1997-2012: a before-and-after study. Lancet Psychiatry. 2016;3(6):526–34.

    Article  Google Scholar 

  3. van Hemert A, Kerkhof A, de Keijser B, Verwey C, van Boven J, Hummelen J, et al Multidisciplinaire richtlijn Diagnostiek en Behandeling van Suicidaal Gedrag [multidisciplinary guideline diagnostics and treatment of suicidal behavior]. Utrecht: De Tijdstroom. 2012.

  4. de Beurs DP, de Groot MH, de Keijser J, van Duijn E, de Winter RF, Kerkhof AJ. Evaluation of benefit to patients of training mental health professionals in suicide guidelines: cluster randomised trial. Br J Psychiatry. 2016;208(5):477–83.

    Article  Google Scholar 

  5. Mokkenstorm J, Franx G, Gilissen R, Kerkhof A, Smit JH. Suicide Prevention Guideline Implementation in Specialist Mental Healthcare Institutions in The Netherlands. Int J Environ Res Public Health. 2018;15(5).

    Article  Google Scholar 

  6. Setkowski K, Mokkenstorm J, van Balkom AJLM, Franx G, Verbeek-Van Noord I, Dongelmans DA, et al. Feasibility and impact of data-driven learning within the suicide prevention action network of thirteen specialist mental healthcare institutions (SUPRANET Care) in the Netherlands: A study protocol. BMJ Open. 2018.

  7. Profit J, Gould JB, Zupancic JA, Stark AR, Wall KM, Kowalkowski MA, et al. Formal selection of measures for a composite index of NICU quality of care: baby-MONITOR. J Perinatol. 2011;31(11):702–10.

    Article  CAS  Google Scholar 

  8. Chen L, Huang LH, Xing MY, Feng ZX, Shao LW, Zhang MY, et al. Using the Delphi method to develop nursing-sensitive quality indicators for the NICU. J Clin Nurs. 2017;26(3–4):502–13.

    Article  Google Scholar 

  9. Koning De J, Smulders A, Klazinga NS. Appraisal of Indicators through Research and Evaluation (AIRE). Amsterdam Medisch Centrum Universiteit van Amsterdam, afdeling Sociale Geneeskunde Amsterdam. 2007.

  10. van de Glind I, Berben S, Zeegers F, Poppen H, Hoogeveen M, Bolt I, et al. A national research agenda for pre-hospital emergency medical services in the Netherlands: a Delphi-study. Scand J Trauma Resusc Emerg Med. 2016;24:2.

    Article  Google Scholar 

  11. Kallen MC, Roos-Blom MJ, Dongelmans DA, Schouten JA, Gude WT, de Jonge E, et al. Development of actionable quality indicators and an action implementation toolbox for appropriate antibiotic use at intensive care units: a modified-RAND Delphi study. PLoS One. 2018;13(11):e0207991.

    Article  CAS  Google Scholar 

  12. Kilbourne AM, Keyser D, Pincus HA. Challenges and opportunities in measuring the quality of mental health care. Can J Psychiatr. 2010;55(9):549–57.

    Article  Google Scholar 

  13. Zalsman G, Hawton K, Wasserman D, van Heeringen K, Arensman E, Sarchiapone M, et al. Suicide prevention strategies revisited: 10-year systematic review. Lancet Psychiatry. 2016;3(7):646–59.

    Article  Google Scholar 

  14. Turecki G, Brent DA. Suicide and suicidal behaviour. Lancet. 2016;387(10024):1227–39.

    Article  Google Scholar 

  15. Meerwijk EL, Parekh A, Oquendo MA, Allen IE, Franck LS, Lee KA. Direct versus indirect psychosocial and behavioural interventions to prevent suicide and suicide attempts: a systematic review and meta-analysis. Lancet Psychiatry. 2016;3(6):544–54.

    Article  Google Scholar 

  16. Wilkinson ST, Ballard ED, Block MH, Mathew SJ, Murrough JW, Feder A, et al. The effect of a single dose of intravenous ketamine on suicidal ideation: a systematic review and individual participant data meta-analysis. Am J Pschiatry. 2018;175(2):150–8.

    Article  Google Scholar 

  17. Brown GK, Jager-Hyman S. Evidence-based psychotherapies for suicide prevention: future directions. Am J Prev Med. 2014;47(3 Suppl 2):S186–94.

    Article  Google Scholar 

  18. Linehan MM, Korslund KE, Harned MS, Gallop RJ, Lungu A, Neacsiu AD, et al. Dialectical behavior therapy for high suicide risk in individuals with borderline personality disorder: a randomized clinical trial and component analysis. JAMA Psychiatry. 2015;72(5):475–82.

    Article  CAS  Google Scholar 

  19. Linehan MM, Comtois KA, Murray AM, Brown MZ, Gallop RJ, Heard HL, et al. Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder. Arch Gen Psychiatry. 2006;63(7):757–66.

    Article  Google Scholar 

  20. Bryan CJ, Mintz J, Clemans TA, Leeson B, Burch TS, Williams SR, et al. Effect of crisis response planning vs. contracts for safety on suicide risk in U.S. Army soldiers: a randomized clinical trial. J Affect Disord. 2017;212:64–72.

    Article  Google Scholar 

  21. Stanley B, Brown GK, Brenner LA, Galfalvy HC, Currier GW, Knox KL, et al. Comparison of the safety planning intervention with follow-up vs usual Care of Suicidal Patients Treated in the emergency department. JAMA Psychiatry. 2018;75(9):894–900.

    Article  Google Scholar 

  22. Brent DA, Oquendo MA, Reynolds CF 3rd. Caring for Suicidal Patients. JAMA Psychiatry. 2019.

  23. While D, Bickley H, Roscoe A, Windfuhr K, Rahman S, Shaw J, et al. Implementation of mental health service recommendations in England and Wales and suicide rates, 1997-2006: a cross-sectional and before-and-after observational study. Lancet. 2012;379(9820):1005–12.

    Article  Google Scholar 

  24. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

    PubMed  CAS  Google Scholar 

  25. Hsu CC, Sandford BA. Minimizing non-response in the Delphi process: how to respond to non-response. Pract Assess Res Eval. 2007;12(17):1–6.

    Google Scholar 

  26. Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12.

    Article  Google Scholar 

  27. Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (standards for QUality improvement reporting excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986–92.

    Article  Google Scholar 

  28. Dewa LH, Murray K, Thibaut B, Ramtale SC, Adam S, Darzi A, et al. Identifying research priorities for patient safety in mental health: an international expert Delphi study. BMJ Open. 2018;8(3):e021361.

    Article  Google Scholar 

  29. Bickley H, Hunt IM, Windfuhr K, Shaw J, Appleby L, Kapur N. Suicide within two weeks of discharge from psychiatric inpatient care: a case-control study. Psychiatr Serv. 2013;64(7):653–9.

    Article  Google Scholar 

  30. Huisman A, Robben PB, Kerkhof AJ. An examination of the Dutch health care Inspectorate's supervision system for suicides of mental health care users. Psychiatr Serv. 2009;60(1):80–5.

    Article  Google Scholar 

  31. Appleby L, Shaw J, Kapur N, Windfur K, Ashton A. Avoidable deaths: five year report by the national confidential inquiry into suicide and homicide by people with mental illness. London: UK department of Health; 2006.

    Google Scholar 

  32. Chung DT, Ryan CJ, Hadzi-Pavlovic D, Singh SP, Stanton C, Large MM. Suicide rates after discharge from psychiatric facilities: a systematic review and meta-analysis. JAMA Psychiatry. 2017;74(7):694–702.

    Article  Google Scholar 

  33. Luxton DD, June JD, Comtois KA. Can postdischarge follow-up contacts prevent suicide and suicidal behavior? A review of the evidence. Crisis. 2013;34(1):32–41.

    Article  Google Scholar 

  34. Mokkenstorm JK, Kerkhof A, Smit JH, ATF B. Is It Rational to pursue zero suicides among patients in health care? Suicide Life Threat Behav. 2017.

    Article  Google Scholar 

  35. Hogan MF, Grumet JG. Suicide prevention: an emerging priority for health care. Health Aff (Millwood). 2016;35(6):1084–90.

    Article  Google Scholar 

  36. National Institute for Health and Care Excellence (NICE). Transition between inpatient mental health settings and community or care home settings. Quality statement 3: Communication on discharge. Quality standard QS159. 2017 Retrieved from https://www.nice.org.uk/guidance/qs159/chapter/Quality-statement-3-Communication-on-discharge. Accessed Sept 2019.

  37. Lang M, Uttaro T, Caine E, Carpinello S, Felton C. Implementing routine suicide risk screening for psychiatric outpatients with serious mental disorders: I. Qualitative Results Arch Suicide Res. 2009;13(2):160–8.

    Article  Google Scholar 

  38. Huisman A, Kerkhof AJFM. The risk of including suicidal patients in RCT studies into the effectiveness of pharmacotherapy and psychotherapy for adult psychiatric disorders and iatrogenic effects as a result of asking about suicide. 2017. https://wwwzonmwnl/nl/actueel/nieuws/detail/item/suicidale-patient-kan-veilig-aan-onderzoek-meedoen/ Accessed Jan 2019.

  39. Chan MKY, Bhatti H, Meader N, Stockton S, Evans J, O'Connor RC, et al. Predicting suicide following self-harm: systematic review of risk factors and risk scales. Br J Psychiatry. 2016;209(4):277–83.

    Article  Google Scholar 

  40. Carter G, Milner A, McGill K, Pirkis J, Kapur N, Spittal MJ. Predicitng suicidal behaviours using clinical instruments: systematic review and meta-analysis of positive predictive values for risk scales. Br J Psychiatry. 2017;210(6):387–95.

    Article  Google Scholar 

  41. Coffey MJ, Coffey CE, Ahmedani BK. Suicide in a health maintenance organization population. JAMA Psychiatry. 2015;72(3):294–6.

    Article  Google Scholar 

  42. National Action Alliance for Suicide Prevention: Transforming Health Systems Initiative Work Group. Recommended standard care for people with suicide risk: Making health care suicide safe. Washington DC: Education Development Center, Inc. 2018.

  43. National Institute for Health and Care Excellence (NICE). Self-harm. Quality statement 6: Risk management plans. Quality standard QS34. 2013 Retrieved from https://www.nice.org.uk/guidance/qs34/chapter/Quality-statement-6-Risk-management-plans. Accessed Sept 2019.

  44. Stanley B, Brown G, Brent DA, Wells K, Poling K, Curry J, et al. Cognitive-behavioral therapy for suicide prevention (CBT-SP): treatment model, feasibility, and acceptability. J Am Acad Child Adolesc Psychiatry. 2009;48(10):1005–13.

    Article  Google Scholar 

  45. Verwey B Don't forget: contributions to the assessment and management of suicide attempters in the general hospital. Doctoral dissertation, University of Leiden. 2006.

  46. Hermens M, van Wetten H, Sinnema H. Kwaliteitsdocument Ketenzorg Suïcidaliteit Utrecht: Trimbos Instituut. 2010.

  47. Hampton T. Depression care effort brings dramatic drop in large HMO population's suicide rate. JAMA. 2010;303(19):1903–5.

    Article  CAS  Google Scholar 

  48. National Institute for Health and Care Excellene (NICE). Suicide prevention. Quality statement 4: Involving family, carers or friends. Quality standard QS189. 2019 Retrieved from https://www.nice.org.uk/guidance/qs189/chapter/Quality-statement-4-Involving-family-carers-or-friends. Accessed Sept 2019.

  49. Barrero SA. Preventing suicide: a resource for the family. Ann General Psychiatry. 2008;7:1.

    Article  Google Scholar 

  50. Wilkins N, Thigpen S, Lockman J, Mackin J, Madden M, Perkins T, et al. Putting program evaluation to work: a framework for creating actionable knowledge for suicide prevention practice. Transl Behav Med. 2013;3(2):149–61.

    Article  Google Scholar 

  51. Trinanes Y, Senra-Rivera C, Seoane-Pesqueira G, Gonzalez-Garcia A, Alvarez-Ariza M, de-Las-Heras-Linero E, et al. Perceived satisfaction and usefulness of suicide prevention information for patients and relatives. Rev Calid Asist. 2014;29(1):36–42.

    Article  CAS  Google Scholar 

Download references

Acknowledgments

The authors would like to thank all suicide experts, members of the patients’ advisory board, experts with experiences in suicidal behavior and healthcare professionals for their involvement and participation in this study. Thanks to the 16 specialist MHIs for their efforts and participation in the SUPRANET network.

Funding

This study is funded by the Ministry of Health Funding-program for Health Care Efficiency Research ZonMw (537001006). ZonMw is not involved in the design of the study and does not participate in the data collection, analysis of the data and writing of the manuscript.

Author information

Authors and Affiliations

  1. 113 Suicide Prevention, Amsterdam, The Netherlands

    Kim Setkowski & Renske Gilissen

  2. Department of Psychiatry, Amsterdam UMC, location VUmc, Amsterdam Public Health research institute and GGZ inGeest, Amsterdam, The Netherlands

    Anton J. L. M. van Balkom

  3. Department of Intensive Care Medicine, Amsterdam UMC, location AMC, Amsterdam, The Netherlands

    Dave A. Dongelmans

  4. National Intensive Care Evaluation (NICE) foundation, Amsterdam, The Netherlands

    Dave A. Dongelmans

Authors
  1. Kim Setkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anton J. L. M. van Balkom
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dave A. Dongelmans
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Renske Gilissen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the design of the study. KS conducted the Delphi study and drafted the manuscript. RG, DD and AJLMvB made critical revisions to and edited the manuscript. All authors contributed to and approved the final manuscript.

Corresponding author

Correspondence to Kim Setkowski.

Ethics declarations

Ethics approval and consent to participate

This study is part of the SUPRANET Care project and has been approved by the Central Committee on Research Involving Human Subjects (CCMO) in the Netherlands. This study does not fall under the scope of the Medical Research Involving Human Subjects Act (WMO). Consent of participants was given by filling in the survey. Data of respondents were anonymized prior to analysis.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1:

Questionnaire round 1 of the Delphi study, to achieve convergence of opinion. Quality indicators were rated by the participants on two selection criteria: relevance and action orientation

Additional file 2:

Questionnaire round 2 of the Delphi study. Quality indicators were rated by the participants on feasibility

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Setkowski, K., van Balkom, A.J.L.M., Dongelmans, D.A. et al. Prioritizing suicide prevention guideline recommendations in specialist mental healthcare: a Delphi study. BMC Psychiatry 20, 55 (2020). https://doi.org/10.1186/s12888-020-2465-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12888-020-2465-0

Keywords

BMC Psychiatry

ISSN: 1471-244X

Contact us